2023-12-04T05:17:50.214 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-04T05:17:50.214 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} --archive /home/teuthworker/archive/vshankar-2023-12-04_05:06:51-fs-wip-vshankar-testing-20231127.102654-distro-default-smithi/7476686 --owner scheduled_vshankar@teuthology --verbose --name vshankar-2023-12-04_05:06:51-fs-wip-vshankar-testing-20231127.102654-distro-default-smithi -- /home/teuthworker/archive/vshankar-2023-12-04_05:06:51-fs-wip-vshankar-testing-20231127.102654-distro-default-smithi/7476686/orig.config.yaml 2023-12-04T05:17:50.266 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T05:17:50.354 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2023-12-04_05:06:51-fs-wip-vshankar-testing-20231127.102654-distro-default-smithi/7476686 branch: wip-vshankar-testing-20231127.102654 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} email: vshankar@redhat.com first_in_suite: false job_id: '7476686' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2023-12-04_05:06:51-fs-wip-vshankar-testing-20231127.102654-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-vshankar-testing-20231127.102654 ceph: cephfs: max_mds: 5 session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: false mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 filestore flush min: 0 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - Replacing daemon mds - SLOW_OPS - slow request sha1: 47c1c547372ed5bff7984647dc200bac0e019caf ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 47c1c547372ed5bff7984647dc200bac0e019caf thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20231127.102654 sha1: 47c1c547372ed5bff7984647dc200bac0e019caf owner: scheduled_vshankar@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 1098 sha1: 47c1c547372ed5bff7984647dc200bac0e019caf sleep_before_teardown: 0 subset: 854/1024 suite: fs suite_branch: wip-vshankar-testing-20231127.102654 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 47c1c547372ed5bff7984647dc200bac0e019caf syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ+YUJ6V8Zs/mK0H0UauHpLsCO+FgY8+C6vlLV1lZ8ShROidb5PMfa1jTYT/99lunhuwhLIQEe6ftNQDd9GOH0I= smithi195.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFxdzf5XteYu1uo6SobO6K4ofYoKZF5RPBVzpjxQRrDSb/joT32TSkCfOjhGbd6T6OSXYnSWiFlLbP63Ry7LmfM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - mds_thrash: null - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-04_05:06:51 tube: smithi use_shaman: true user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-04T05:17:50.354 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa; will attempt to use it 2023-12-04T05:17:50.355 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks 2023-12-04T05:17:50.355 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-04T05:17:50.357 INFO:teuthology.task.internal:Checking packages... 2023-12-04T05:17:50.382 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '47c1c547372ed5bff7984647dc200bac0e019caf' 2023-12-04T05:17:50.382 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-04T05:17:50.382 INFO:teuthology.packaging:ref: None 2023-12-04T05:17:50.382 INFO:teuthology.packaging:tag: None 2023-12-04T05:17:50.382 INFO:teuthology.packaging:branch: wip-vshankar-testing-20231127.102654 2023-12-04T05:17:50.382 INFO:teuthology.packaging:sha1: 47c1c547372ed5bff7984647dc200bac0e019caf 2023-12-04T05:17:50.383 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-vshankar-testing-20231127.102654 2023-12-04T05:17:50.520 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7531-g47c1c547-1jammy 2023-12-04T05:17:50.520 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-04T05:17:50.526 INFO:teuthology.task.internal:no buildpackages task found 2023-12-04T05:17:50.526 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-04T05:17:50.533 INFO:teuthology.task.internal:Saving configuration 2023-12-04T05:17:50.554 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-04T05:17:50.560 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T05:17:50.579 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-12-04_05:06:51-fs-wip-vshankar-testing-20231127.102654-distro-default-smithi/7476686', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 05:09:04.580894', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILZidqXeBEfJ3wN8QJeiY1Iiki9rSFvZDaQk0u7ts6q5'} 2023-12-04T05:17:50.597 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-12-04_05:06:51-fs-wip-vshankar-testing-20231127.102654-distro-default-smithi/7476686', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 05:09:04.582334', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMiP8cRvyQFwCWH0i7Q1qAXrSqqVrHybK2B0yXtdXxWb'} 2023-12-04T05:17:50.597 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-04T05:17:50.603 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-12-04T05:17:50.603 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-12-04T05:17:50.603 INFO:teuthology.run_tasks:Running task console_log... 2023-12-04T05:17:50.668 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa68fe97040>, signals=[15]) 2023-12-04T05:17:50.668 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-04T05:17:50.677 INFO:teuthology.task.internal:Opening connections... 2023-12-04T05:17:50.678 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2023-12-04T05:17:50.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T05:17:50.762 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi195.front.sepia.ceph.com 2023-12-04T05:17:50.763 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T05:17:50.834 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-04T05:17:50.841 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2023-12-04T05:17:50.846 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2023-12-04T05:17:50.846 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2023-12-04T05:17:50.892 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-12-04T05:17:50.892 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="Ubuntu" 2023-12-04T05:17:50.892 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="22.04" 2023-12-04T05:17:50.893 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-12-04T05:17:50.893 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_CODENAME=jammy 2023-12-04T05:17:50.893 INFO:teuthology.orchestra.run.smithi107.stdout:ID=ubuntu 2023-12-04T05:17:50.893 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE=debian 2023-12-04T05:17:50.893 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://www.ubuntu.com/" 2023-12-04T05:17:50.893 INFO:teuthology.orchestra.run.smithi107.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-12-04T05:17:50.893 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-12-04T05:17:50.893 INFO:teuthology.orchestra.run.smithi107.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-12-04T05:17:50.893 INFO:teuthology.orchestra.run.smithi107.stdout:UBUNTU_CODENAME=jammy 2023-12-04T05:17:50.894 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2023-12-04T05:17:50.916 DEBUG:teuthology.orchestra.run.smithi195:> uname -m 2023-12-04T05:17:50.921 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2023-12-04T05:17:50.921 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release 2023-12-04T05:17:50.966 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-12-04T05:17:50.966 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="Ubuntu" 2023-12-04T05:17:50.966 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="22.04" 2023-12-04T05:17:50.966 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-12-04T05:17:50.966 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_CODENAME=jammy 2023-12-04T05:17:50.966 INFO:teuthology.orchestra.run.smithi195.stdout:ID=ubuntu 2023-12-04T05:17:50.966 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE=debian 2023-12-04T05:17:50.966 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://www.ubuntu.com/" 2023-12-04T05:17:50.967 INFO:teuthology.orchestra.run.smithi195.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-12-04T05:17:50.967 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-12-04T05:17:50.967 INFO:teuthology.orchestra.run.smithi195.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-12-04T05:17:50.967 INFO:teuthology.orchestra.run.smithi195.stdout:UBUNTU_CODENAME=jammy 2023-12-04T05:17:50.967 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2023-12-04T05:17:50.982 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-04T05:17:50.992 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-04T05:17:50.997 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-04T05:17:50.998 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2023-12-04T05:17:51.000 DEBUG:teuthology.orchestra.run.smithi195:> test '!' -e /home/ubuntu/cephtest 2023-12-04T05:17:51.013 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-04T05:17:51.019 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-04T05:17:51.020 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2023-12-04T05:17:51.046 DEBUG:teuthology.orchestra.run.smithi195:> test -z $(ls -A /var/lib/ceph) 2023-12-04T05:17:51.135 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-04T05:17:51.178 INFO:teuthology.run_tasks:Running task kernel... 2023-12-04T05:17:51.198 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-04T05:17:51.198 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-04T05:17:51.199 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T05:17:51.199 DEBUG:teuthology.orchestra.run.smithi195:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T05:17:51.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T05:17:51.204 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2023-12-04T05:17:51.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T05:17:51.205 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2023-12-04T05:17:51.250 INFO:teuthology.orchestra.run.smithi107.stdout:5.15.0-56-generic 2023-12-04T05:17:51.250 INFO:teuthology.task.kernel:Running kernel on smithi107: 5.15.0-56-generic 2023-12-04T05:17:51.250 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt-get clean 2023-12-04T05:17:51.255 INFO:teuthology.orchestra.run.smithi195.stdout:5.15.0-56-generic 2023-12-04T05:17:51.255 INFO:teuthology.task.kernel:Running kernel on smithi195: 5.15.0-56-generic 2023-12-04T05:17:51.256 DEBUG:teuthology.orchestra.run.smithi195:> sudo apt-get clean 2023-12-04T05:17:51.367 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt-get update 2023-12-04T05:17:51.377 DEBUG:teuthology.orchestra.run.smithi195:> sudo apt-get update 2023-12-04T05:17:51.494 INFO:teuthology.orchestra.run.smithi107.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-12-04T05:17:51.603 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-04T05:17:51.648 INFO:teuthology.orchestra.run.smithi195.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-12-04T05:17:51.685 INFO:teuthology.orchestra.run.smithi107.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-12-04T05:17:51.712 INFO:teuthology.orchestra.run.smithi107.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [373 kB] 2023-12-04T05:17:51.785 INFO:teuthology.orchestra.run.smithi107.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,002 kB] 2023-12-04T05:17:51.823 INFO:teuthology.orchestra.run.smithi107.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [193 kB] 2023-12-04T05:17:51.827 INFO:teuthology.orchestra.run.smithi107.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-12-04T05:17:51.827 INFO:teuthology.orchestra.run.smithi107.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,164 kB] 2023-12-04T05:17:51.849 INFO:teuthology.orchestra.run.smithi107.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.4 kB] 2023-12-04T05:17:51.850 INFO:teuthology.orchestra.run.smithi107.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [189 kB] 2023-12-04T05:17:51.854 INFO:teuthology.orchestra.run.smithi107.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2023-12-04T05:17:51.854 INFO:teuthology.orchestra.run.smithi107.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [807 kB] 2023-12-04T05:17:51.866 INFO:teuthology.orchestra.run.smithi107.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [573 kB] 2023-12-04T05:17:51.873 INFO:teuthology.orchestra.run.smithi107.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [150 kB] 2023-12-04T05:17:51.874 INFO:teuthology.orchestra.run.smithi107.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-12-04T05:17:51.898 INFO:teuthology.orchestra.run.smithi107.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-12-04T05:17:51.899 INFO:teuthology.orchestra.run.smithi107.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-12-04T05:17:51.899 INFO:teuthology.orchestra.run.smithi107.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-12-04T05:17:51.899 INFO:teuthology.orchestra.run.smithi107.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-12-04T05:17:52.050 INFO:teuthology.orchestra.run.smithi107.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-12-04T05:17:52.136 INFO:teuthology.orchestra.run.smithi195.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,002 kB] 2023-12-04T05:17:52.189 INFO:teuthology.orchestra.run.smithi107.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,211 kB] 2023-12-04T05:17:52.472 INFO:teuthology.orchestra.run.smithi107.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [537 kB] 2023-12-04T05:17:52.509 INFO:teuthology.orchestra.run.smithi107.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [253 kB] 2023-12-04T05:17:52.521 INFO:teuthology.orchestra.run.smithi195.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [373 kB] 2023-12-04T05:17:52.527 INFO:teuthology.orchestra.run.smithi107.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2023-12-04T05:17:52.527 INFO:teuthology.orchestra.run.smithi107.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,185 kB] 2023-12-04T05:17:52.541 INFO:teuthology.orchestra.run.smithi195.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [193 kB] 2023-12-04T05:17:52.552 INFO:teuthology.orchestra.run.smithi195.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-12-04T05:17:52.553 INFO:teuthology.orchestra.run.smithi195.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,164 kB] 2023-12-04T05:17:52.583 INFO:teuthology.orchestra.run.smithi107.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.8 kB] 2023-12-04T05:17:52.584 INFO:teuthology.orchestra.run.smithi107.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [192 kB] 2023-12-04T05:17:52.590 INFO:teuthology.orchestra.run.smithi107.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2023-12-04T05:17:52.639 INFO:teuthology.orchestra.run.smithi195.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.4 kB] 2023-12-04T05:17:52.640 INFO:teuthology.orchestra.run.smithi195.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [189 kB] 2023-12-04T05:17:52.651 INFO:teuthology.orchestra.run.smithi195.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2023-12-04T05:17:52.651 INFO:teuthology.orchestra.run.smithi195.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [807 kB] 2023-12-04T05:17:52.705 INFO:teuthology.orchestra.run.smithi195.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [573 kB] 2023-12-04T05:17:52.754 INFO:teuthology.orchestra.run.smithi107.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [671 kB] 2023-12-04T05:17:52.878 INFO:teuthology.orchestra.run.smithi195.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [150 kB] 2023-12-04T05:17:53.272 INFO:teuthology.orchestra.run.smithi195.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-12-04T05:17:53.278 INFO:teuthology.orchestra.run.smithi195.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-12-04T05:17:53.293 INFO:teuthology.orchestra.run.smithi195.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-12-04T05:17:53.294 INFO:teuthology.orchestra.run.smithi195.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-12-04T05:17:53.297 INFO:teuthology.orchestra.run.smithi195.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-12-04T05:17:53.328 INFO:teuthology.orchestra.run.smithi107.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,010 kB] 2023-12-04T05:17:53.437 INFO:teuthology.orchestra.run.smithi107.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [224 kB] 2023-12-04T05:17:53.456 INFO:teuthology.orchestra.run.smithi107.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2023-12-04T05:17:53.459 INFO:teuthology.orchestra.run.smithi107.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-12-04T05:17:53.459 INFO:teuthology.orchestra.run.smithi107.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-12-04T05:17:53.462 INFO:teuthology.orchestra.run.smithi107.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-12-04T05:17:53.463 INFO:teuthology.orchestra.run.smithi107.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-12-04T05:17:53.463 INFO:teuthology.orchestra.run.smithi107.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-12-04T05:17:53.468 INFO:teuthology.orchestra.run.smithi107.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-12-04T05:17:53.472 INFO:teuthology.orchestra.run.smithi107.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-12-04T05:17:53.637 INFO:teuthology.orchestra.run.smithi107.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-12-04T05:17:53.719 INFO:teuthology.orchestra.run.smithi107.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.8 kB] 2023-12-04T05:17:53.804 INFO:teuthology.orchestra.run.smithi107.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-12-04T05:17:53.885 INFO:teuthology.orchestra.run.smithi107.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2023-12-04T05:17:53.916 INFO:teuthology.orchestra.run.smithi107.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-12-04T05:17:58.650 INFO:teuthology.orchestra.run.smithi107.stdout:Fetched 10.5 MB in 2s (4,215 kB/s) 2023-12-04T05:17:58.763 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:18 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-04T05:17:58.780 INFO:teuthology.orchestra.run.smithi195.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-12-04T05:17:58.854 INFO:teuthology.orchestra.run.smithi195.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-12-04T05:17:58.950 INFO:teuthology.orchestra.run.smithi195.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,211 kB] 2023-12-04T05:17:59.026 INFO:teuthology.orchestra.run.smithi195.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [537 kB] 2023-12-04T05:17:59.031 INFO:teuthology.orchestra.run.smithi195.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [253 kB] 2023-12-04T05:17:59.033 INFO:teuthology.orchestra.run.smithi195.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2023-12-04T05:17:59.033 INFO:teuthology.orchestra.run.smithi195.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.8 kB] 2023-12-04T05:17:59.034 INFO:teuthology.orchestra.run.smithi195.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,185 kB] 2023-12-04T05:17:59.049 INFO:teuthology.orchestra.run.smithi195.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [192 kB] 2023-12-04T05:17:59.050 INFO:teuthology.orchestra.run.smithi195.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2023-12-04T05:17:59.051 INFO:teuthology.orchestra.run.smithi195.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [671 kB] 2023-12-04T05:17:59.056 INFO:teuthology.orchestra.run.smithi195.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,010 kB] 2023-12-04T05:17:59.064 INFO:teuthology.orchestra.run.smithi195.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [224 kB] 2023-12-04T05:17:59.066 INFO:teuthology.orchestra.run.smithi195.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2023-12-04T05:17:59.067 INFO:teuthology.orchestra.run.smithi195.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-12-04T05:17:59.067 INFO:teuthology.orchestra.run.smithi195.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-12-04T05:17:59.068 INFO:teuthology.orchestra.run.smithi195.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-12-04T05:17:59.083 INFO:teuthology.orchestra.run.smithi195.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-12-04T05:17:59.083 INFO:teuthology.orchestra.run.smithi195.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-12-04T05:17:59.084 INFO:teuthology.orchestra.run.smithi195.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-12-04T05:17:59.085 INFO:teuthology.orchestra.run.smithi195.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-12-04T05:17:59.085 INFO:teuthology.orchestra.run.smithi195.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-12-04T05:17:59.085 INFO:teuthology.orchestra.run.smithi195.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-12-04T05:17:59.085 INFO:teuthology.orchestra.run.smithi195.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.8 kB] 2023-12-04T05:17:59.101 INFO:teuthology.orchestra.run.smithi195.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2023-12-04T05:17:59.101 INFO:teuthology.orchestra.run.smithi195.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-12-04T05:17:59.968 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2023-12-04T05:17:59.986 DEBUG:teuthology.orchestra.run.smithi107:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-04T05:18:00.049 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2023-12-04T05:18:00.237 INFO:teuthology.orchestra.run.smithi107.stdout:Building dependency tree... 2023-12-04T05:18:00.238 INFO:teuthology.orchestra.run.smithi107.stdout:Reading state information... 2023-12-04T05:18:00.404 INFO:teuthology.orchestra.run.smithi107.stdout:The following packages were automatically installed and are no longer required: 2023-12-04T05:18:00.404 INFO:teuthology.orchestra.run.smithi107.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-04T05:18:00.404 INFO:teuthology.orchestra.run.smithi107.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-04T05:18:00.405 INFO:teuthology.orchestra.run.smithi107.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-04T05:18:00.405 INFO:teuthology.orchestra.run.smithi107.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-12-04T05:18:00.405 INFO:teuthology.orchestra.run.smithi107.stdout: python2.7-minimal 2023-12-04T05:18:00.405 INFO:teuthology.orchestra.run.smithi107.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-04T05:18:00.406 INFO:teuthology.orchestra.run.smithi107.stdout:The following additional packages will be installed: 2023-12-04T05:18:00.406 INFO:teuthology.orchestra.run.smithi107.stdout: linux-generic linux-headers-5.15.0-89 linux-headers-5.15.0-89-generic 2023-12-04T05:18:00.406 INFO:teuthology.orchestra.run.smithi107.stdout: linux-headers-generic linux-image-5.15.0-89-generic 2023-12-04T05:18:00.406 INFO:teuthology.orchestra.run.smithi107.stdout: linux-modules-5.15.0-89-generic linux-modules-extra-5.15.0-89-generic 2023-12-04T05:18:00.407 INFO:teuthology.orchestra.run.smithi107.stdout:Suggested packages: 2023-12-04T05:18:00.407 INFO:teuthology.orchestra.run.smithi107.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-12-04T05:18:00.407 INFO:teuthology.orchestra.run.smithi107.stdout:Recommended packages: 2023-12-04T05:18:00.407 INFO:teuthology.orchestra.run.smithi107.stdout: thermald 2023-12-04T05:18:00.454 INFO:teuthology.orchestra.run.smithi107.stdout:The following NEW packages will be installed: 2023-12-04T05:18:00.455 INFO:teuthology.orchestra.run.smithi107.stdout: linux-headers-5.15.0-89 linux-headers-5.15.0-89-generic 2023-12-04T05:18:00.455 INFO:teuthology.orchestra.run.smithi107.stdout: linux-image-5.15.0-89-generic linux-modules-5.15.0-89-generic 2023-12-04T05:18:00.455 INFO:teuthology.orchestra.run.smithi107.stdout: linux-modules-extra-5.15.0-89-generic 2023-12-04T05:18:00.455 INFO:teuthology.orchestra.run.smithi107.stdout:The following packages will be upgraded: 2023-12-04T05:18:00.456 INFO:teuthology.orchestra.run.smithi107.stdout: linux-generic linux-headers-generic linux-image-generic 2023-12-04T05:18:00.505 INFO:teuthology.orchestra.run.smithi107.stdout:3 upgraded, 5 newly installed, 0 to remove and 274 not upgraded. 2023-12-04T05:18:00.505 INFO:teuthology.orchestra.run.smithi107.stdout:Need to get 113 MB of archives. 2023-12-04T05:18:00.506 INFO:teuthology.orchestra.run.smithi107.stdout:After this operation, 582 MB of additional disk space will be used. 2023-12-04T05:18:00.506 INFO:teuthology.orchestra.run.smithi107.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-89-generic amd64 5.15.0-89.99 [22.4 MB] 2023-12-04T05:18:00.909 INFO:teuthology.orchestra.run.smithi107.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-89-generic amd64 5.15.0-89.99 [11.5 MB] 2023-12-04T05:18:00.998 INFO:teuthology.orchestra.run.smithi107.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-89-generic amd64 5.15.0-89.99 [63.9 MB] 2023-12-04T05:18:01.808 INFO:teuthology.orchestra.run.smithi107.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.89.86 [1,698 B] 2023-12-04T05:18:01.809 INFO:teuthology.orchestra.run.smithi107.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.89.86 [2,492 B] 2023-12-04T05:18:01.809 INFO:teuthology.orchestra.run.smithi107.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-89 all 5.15.0-89.99 [12.3 MB] 2023-12-04T05:18:01.927 INFO:teuthology.orchestra.run.smithi107.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-89-generic amd64 5.15.0-89.99 [2,885 kB] 2023-12-04T05:18:01.948 INFO:teuthology.orchestra.run.smithi107.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.89.86 [2,336 B] 2023-12-04T05:18:02.342 INFO:teuthology.orchestra.run.smithi107.stdout:Fetched 113 MB in 1s (76.1 MB/s) 2023-12-04T05:18:02.503 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package linux-modules-5.15.0-89-generic. 2023-12-04T05:18:03.887 INFO:teuthology.orchestra.run.smithi195.stdout:Fetched 10.5 MB in 8s (1,285 kB/s) 2023-12-04T05:18:04.552 INFO:teuthology.orchestra.run.smithi107.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-12-04T05:18:04.557 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../0-linux-modules-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-12-04T05:18:04.768 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-modules-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-04T05:18:05.195 INFO:teuthology.orchestra.run.smithi195.stdout:Reading package lists... 2023-12-04T05:18:05.212 DEBUG:teuthology.orchestra.run.smithi195:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-04T05:18:05.273 INFO:teuthology.orchestra.run.smithi195.stdout:Reading package lists... 2023-12-04T05:18:05.471 INFO:teuthology.orchestra.run.smithi195.stdout:Building dependency tree... 2023-12-04T05:18:05.472 INFO:teuthology.orchestra.run.smithi195.stdout:Reading state information... 2023-12-04T05:18:05.640 INFO:teuthology.orchestra.run.smithi195.stdout:The following packages were automatically installed and are no longer required: 2023-12-04T05:18:05.640 INFO:teuthology.orchestra.run.smithi195.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-04T05:18:05.640 INFO:teuthology.orchestra.run.smithi195.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-04T05:18:05.640 INFO:teuthology.orchestra.run.smithi195.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-04T05:18:05.640 INFO:teuthology.orchestra.run.smithi195.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-12-04T05:18:05.641 INFO:teuthology.orchestra.run.smithi195.stdout: python2.7-minimal 2023-12-04T05:18:05.641 INFO:teuthology.orchestra.run.smithi195.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-04T05:18:05.642 INFO:teuthology.orchestra.run.smithi195.stdout:The following additional packages will be installed: 2023-12-04T05:18:05.642 INFO:teuthology.orchestra.run.smithi195.stdout: linux-generic linux-headers-5.15.0-89 linux-headers-5.15.0-89-generic 2023-12-04T05:18:05.642 INFO:teuthology.orchestra.run.smithi195.stdout: linux-headers-generic linux-image-5.15.0-89-generic 2023-12-04T05:18:05.642 INFO:teuthology.orchestra.run.smithi195.stdout: linux-modules-5.15.0-89-generic linux-modules-extra-5.15.0-89-generic 2023-12-04T05:18:05.644 INFO:teuthology.orchestra.run.smithi195.stdout:Suggested packages: 2023-12-04T05:18:05.644 INFO:teuthology.orchestra.run.smithi195.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-12-04T05:18:05.644 INFO:teuthology.orchestra.run.smithi195.stdout:Recommended packages: 2023-12-04T05:18:05.644 INFO:teuthology.orchestra.run.smithi195.stdout: thermald 2023-12-04T05:18:05.696 INFO:teuthology.orchestra.run.smithi195.stdout:The following NEW packages will be installed: 2023-12-04T05:18:05.696 INFO:teuthology.orchestra.run.smithi195.stdout: linux-headers-5.15.0-89 linux-headers-5.15.0-89-generic 2023-12-04T05:18:05.696 INFO:teuthology.orchestra.run.smithi195.stdout: linux-image-5.15.0-89-generic linux-modules-5.15.0-89-generic 2023-12-04T05:18:05.696 INFO:teuthology.orchestra.run.smithi195.stdout: linux-modules-extra-5.15.0-89-generic 2023-12-04T05:18:05.697 INFO:teuthology.orchestra.run.smithi195.stdout:The following packages will be upgraded: 2023-12-04T05:18:05.698 INFO:teuthology.orchestra.run.smithi195.stdout: linux-generic linux-headers-generic linux-image-generic 2023-12-04T05:18:05.877 INFO:teuthology.orchestra.run.smithi195.stdout:3 upgraded, 5 newly installed, 0 to remove and 274 not upgraded. 2023-12-04T05:18:05.877 INFO:teuthology.orchestra.run.smithi195.stdout:Need to get 113 MB of archives. 2023-12-04T05:18:05.877 INFO:teuthology.orchestra.run.smithi195.stdout:After this operation, 582 MB of additional disk space will be used. 2023-12-04T05:18:05.877 INFO:teuthology.orchestra.run.smithi195.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-89-generic amd64 5.15.0-89.99 [22.4 MB] 2023-12-04T05:18:07.277 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package linux-image-5.15.0-89-generic. 2023-12-04T05:18:07.299 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../1-linux-image-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-12-04T05:18:07.371 INFO:teuthology.orchestra.run.smithi195.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-89-generic amd64 5.15.0-89.99 [11.5 MB] 2023-12-04T05:18:07.392 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-image-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-04T05:18:07.724 INFO:teuthology.orchestra.run.smithi195.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-89-generic amd64 5.15.0-89.99 [63.9 MB] 2023-12-04T05:18:07.822 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-89-generic. 2023-12-04T05:18:07.843 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-12-04T05:18:07.880 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-modules-extra-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-04T05:18:09.911 INFO:teuthology.orchestra.run.smithi195.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.89.86 [1,698 B] 2023-12-04T05:18:09.912 INFO:teuthology.orchestra.run.smithi195.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.89.86 [2,492 B] 2023-12-04T05:18:09.912 INFO:teuthology.orchestra.run.smithi195.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-89 all 5.15.0-89.99 [12.3 MB] 2023-12-04T05:18:10.578 INFO:teuthology.orchestra.run.smithi195.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-89-generic amd64 5.15.0-89.99 [2,885 kB] 2023-12-04T05:18:10.708 INFO:teuthology.orchestra.run.smithi195.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.89.86 [2,336 B] 2023-12-04T05:18:11.107 INFO:teuthology.orchestra.run.smithi195.stdout:Fetched 113 MB in 5s (22.6 MB/s) 2023-12-04T05:18:11.274 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package linux-modules-5.15.0-89-generic. 2023-12-04T05:18:13.264 INFO:teuthology.orchestra.run.smithi195.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-12-04T05:18:13.268 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../0-linux-modules-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-12-04T05:18:13.346 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking linux-modules-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-04T05:18:14.396 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../3-linux-generic_5.15.0.89.86_amd64.deb ... 2023-12-04T05:18:14.499 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-generic (5.15.0.89.86) over (5.15.0.56.54) ... 2023-12-04T05:18:14.866 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.89.86_amd64.deb ... 2023-12-04T05:18:14.994 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-image-generic (5.15.0.89.86) over (5.15.0.56.54) ... 2023-12-04T05:18:15.322 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package linux-headers-5.15.0-89. 2023-12-04T05:18:15.333 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../5-linux-headers-5.15.0-89_5.15.0-89.99_all.deb ... 2023-12-04T05:18:15.372 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-headers-5.15.0-89 (5.15.0-89.99) ... 2023-12-04T05:18:15.556 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package linux-image-5.15.0-89-generic. 2023-12-04T05:18:15.577 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../1-linux-image-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-12-04T05:18:15.668 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking linux-image-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-04T05:18:16.084 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-89-generic. 2023-12-04T05:18:16.104 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-12-04T05:18:16.142 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking linux-modules-extra-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-04T05:18:21.079 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package linux-headers-5.15.0-89-generic. 2023-12-04T05:18:21.116 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../6-linux-headers-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-12-04T05:18:21.154 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-headers-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-04T05:18:22.009 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../3-linux-generic_5.15.0.89.86_amd64.deb ... 2023-12-04T05:18:22.137 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking linux-generic (5.15.0.89.86) over (5.15.0.56.54) ... 2023-12-04T05:18:22.476 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.89.86_amd64.deb ... 2023-12-04T05:18:22.598 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking linux-image-generic (5.15.0.89.86) over (5.15.0.56.54) ... 2023-12-04T05:18:22.859 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package linux-headers-5.15.0-89. 2023-12-04T05:18:22.872 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../5-linux-headers-5.15.0-89_5.15.0-89.99_all.deb ... 2023-12-04T05:18:22.917 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking linux-headers-5.15.0-89 (5.15.0-89.99) ... 2023-12-04T05:18:23.858 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.89.86_amd64.deb ... 2023-12-04T05:18:23.955 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-headers-generic (5.15.0.89.86) over (5.15.0.56.54) ... 2023-12-04T05:18:24.304 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-headers-5.15.0-89 (5.15.0-89.99) ... 2023-12-04T05:18:24.426 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-headers-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-04T05:18:24.577 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-headers-generic (5.15.0.89.86) ... 2023-12-04T05:18:24.707 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-image-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-04T05:18:26.042 INFO:teuthology.orchestra.run.smithi107.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-12-04T05:18:26.042 INFO:teuthology.orchestra.run.smithi107.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-12-04T05:18:26.042 INFO:teuthology.orchestra.run.smithi107.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-89-generic 2023-12-04T05:18:26.042 INFO:teuthology.orchestra.run.smithi107.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-89-generic 2023-12-04T05:18:26.207 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-modules-extra-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-04T05:18:27.792 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-image-generic (5.15.0.89.86) ... 2023-12-04T05:18:27.918 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-generic (5.15.0.89.86) ... 2023-12-04T05:18:28.060 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-modules-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-04T05:18:28.812 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package linux-headers-5.15.0-89-generic. 2023-12-04T05:18:28.849 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../6-linux-headers-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-12-04T05:18:28.887 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking linux-headers-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-04T05:18:29.487 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for linux-image-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-04T05:18:29.545 INFO:teuthology.orchestra.run.smithi107.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-12-04T05:18:29.545 INFO:teuthology.orchestra.run.smithi107.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-89-generic 2023-12-04T05:18:31.680 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.89.86_amd64.deb ... 2023-12-04T05:18:31.806 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking linux-headers-generic (5.15.0.89.86) over (5.15.0.56.54) ... 2023-12-04T05:18:32.156 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up linux-headers-5.15.0-89 (5.15.0-89.99) ... 2023-12-04T05:18:32.278 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up linux-headers-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-04T05:18:32.395 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up linux-headers-generic (5.15.0.89.86) ... 2023-12-04T05:18:32.500 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up linux-image-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-04T05:18:33.822 INFO:teuthology.orchestra.run.smithi195.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-12-04T05:18:33.822 INFO:teuthology.orchestra.run.smithi195.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-12-04T05:18:33.822 INFO:teuthology.orchestra.run.smithi195.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-89-generic 2023-12-04T05:18:33.822 INFO:teuthology.orchestra.run.smithi195.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-89-generic 2023-12-04T05:18:33.984 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up linux-modules-extra-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-04T05:18:35.369 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up linux-image-generic (5.15.0.89.86) ... 2023-12-04T05:18:35.495 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up linux-generic (5.15.0.89.86) ... 2023-12-04T05:18:35.612 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up linux-modules-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-04T05:18:37.439 INFO:teuthology.orchestra.run.smithi195.stdout:Processing triggers for linux-image-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-04T05:18:37.507 INFO:teuthology.orchestra.run.smithi195.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-12-04T05:18:37.507 INFO:teuthology.orchestra.run.smithi195.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-89-generic 2023-12-04T05:18:43.845 INFO:teuthology.orchestra.run.smithi107.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-12-04T05:18:43.845 INFO:teuthology.orchestra.run.smithi107.stdout:Sourcing file `/etc/default/grub' 2023-12-04T05:18:43.873 INFO:teuthology.orchestra.run.smithi107.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-04T05:18:43.902 INFO:teuthology.orchestra.run.smithi107.stdout:Generating grub configuration file ... 2023-12-04T05:18:44.207 INFO:teuthology.orchestra.run.smithi107.stdout:Found linux image: /boot/vmlinuz-5.15.0-89-generic 2023-12-04T05:18:44.226 INFO:teuthology.orchestra.run.smithi107.stdout:Found initrd image: /boot/initrd.img-5.15.0-89-generic 2023-12-04T05:18:44.539 INFO:teuthology.orchestra.run.smithi107.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-12-04T05:18:44.544 INFO:teuthology.orchestra.run.smithi107.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-12-04T05:18:44.617 INFO:teuthology.orchestra.run.smithi107.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-12-04T05:18:44.623 INFO:teuthology.orchestra.run.smithi107.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-12-04T05:18:44.788 INFO:teuthology.orchestra.run.smithi107.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-12-04T05:18:44.788 INFO:teuthology.orchestra.run.smithi107.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-12-04T05:18:44.788 INFO:teuthology.orchestra.run.smithi107.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-12-04T05:18:44.838 INFO:teuthology.orchestra.run.smithi107.stdout:done 2023-12-04T05:18:45.423 DEBUG:teuthology.orchestra.run.smithi107:> dpkg -s linux-image-generic 2023-12-04T05:18:45.493 INFO:teuthology.orchestra.run.smithi107.stdout:Package: linux-image-generic 2023-12-04T05:18:45.493 INFO:teuthology.orchestra.run.smithi107.stdout:Status: install ok installed 2023-12-04T05:18:45.493 INFO:teuthology.orchestra.run.smithi107.stdout:Priority: optional 2023-12-04T05:18:45.493 INFO:teuthology.orchestra.run.smithi107.stdout:Section: kernel 2023-12-04T05:18:45.493 INFO:teuthology.orchestra.run.smithi107.stdout:Installed-Size: 20 2023-12-04T05:18:45.493 INFO:teuthology.orchestra.run.smithi107.stdout:Maintainer: Ubuntu Kernel Team 2023-12-04T05:18:45.493 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: amd64 2023-12-04T05:18:45.493 INFO:teuthology.orchestra.run.smithi107.stdout:Source: linux-meta 2023-12-04T05:18:45.493 INFO:teuthology.orchestra.run.smithi107.stdout:Version: 5.15.0.89.86 2023-12-04T05:18:45.494 INFO:teuthology.orchestra.run.smithi107.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-89), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-12-04T05:18:45.494 INFO:teuthology.orchestra.run.smithi107.stdout:Depends: linux-image-5.15.0-89-generic, linux-modules-extra-5.15.0-89-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-04T05:18:45.494 INFO:teuthology.orchestra.run.smithi107.stdout:Recommends: thermald 2023-12-04T05:18:45.494 INFO:teuthology.orchestra.run.smithi107.stdout:Description: Generic Linux kernel image 2023-12-04T05:18:45.494 INFO:teuthology.orchestra.run.smithi107.stdout: This package will always depend on the latest generic kernel image 2023-12-04T05:18:45.494 INFO:teuthology.orchestra.run.smithi107.stdout: available. 2023-12-04T05:18:45.494 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-89-generic 2023-12-04T05:18:45.494 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T05:18:45.494 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T05:18:45.494 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-04T05:18:45.494 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi107.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T05:18:45.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt-get clean 2023-12-04T05:18:45.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt-get update 2023-12-04T05:18:45.921 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-04T05:18:45.921 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-12-04T05:18:45.980 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-12-04T05:18:46.062 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-12-04T05:18:47.242 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2023-12-04T05:18:47.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-04T05:18:47.318 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2023-12-04T05:18:47.507 INFO:teuthology.orchestra.run.smithi107.stdout:Building dependency tree... 2023-12-04T05:18:47.508 INFO:teuthology.orchestra.run.smithi107.stdout:Reading state information... 2023-12-04T05:18:47.675 INFO:teuthology.orchestra.run.smithi107.stdout:linux-image-generic is already the newest version (5.15.0.89.86). 2023-12-04T05:18:47.675 INFO:teuthology.orchestra.run.smithi107.stdout:The following packages were automatically installed and are no longer required: 2023-12-04T05:18:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-04T05:18:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-04T05:18:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-04T05:18:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-04T05:18:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-04T05:18:47.676 INFO:teuthology.orchestra.run.smithi107.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-04T05:18:47.718 INFO:teuthology.orchestra.run.smithi107.stdout:0 upgraded, 0 newly installed, 0 to remove and 274 not upgraded. 2023-12-04T05:18:47.719 DEBUG:teuthology.orchestra.run.smithi107:> dpkg -s linux-image-generic 2023-12-04T05:18:47.735 INFO:teuthology.orchestra.run.smithi107.stdout:Package: linux-image-generic 2023-12-04T05:18:47.735 INFO:teuthology.orchestra.run.smithi107.stdout:Status: install ok installed 2023-12-04T05:18:47.735 INFO:teuthology.orchestra.run.smithi107.stdout:Priority: optional 2023-12-04T05:18:47.735 INFO:teuthology.orchestra.run.smithi107.stdout:Section: kernel 2023-12-04T05:18:47.735 INFO:teuthology.orchestra.run.smithi107.stdout:Installed-Size: 20 2023-12-04T05:18:47.735 INFO:teuthology.orchestra.run.smithi107.stdout:Maintainer: Ubuntu Kernel Team 2023-12-04T05:18:47.735 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: amd64 2023-12-04T05:18:47.735 INFO:teuthology.orchestra.run.smithi107.stdout:Source: linux-meta 2023-12-04T05:18:47.735 INFO:teuthology.orchestra.run.smithi107.stdout:Version: 5.15.0.89.86 2023-12-04T05:18:47.735 INFO:teuthology.orchestra.run.smithi107.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-89), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-12-04T05:18:47.735 INFO:teuthology.orchestra.run.smithi107.stdout:Depends: linux-image-5.15.0-89-generic, linux-modules-extra-5.15.0-89-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-04T05:18:47.735 INFO:teuthology.orchestra.run.smithi107.stdout:Recommends: thermald 2023-12-04T05:18:47.736 INFO:teuthology.orchestra.run.smithi107.stdout:Description: Generic Linux kernel image 2023-12-04T05:18:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: This package will always depend on the latest generic kernel image 2023-12-04T05:18:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: available. 2023-12-04T05:18:47.736 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2023-12-04T05:18:47.781 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.tXsPznABtK 2023-12-04T05:18:47.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo cp /boot/grub/grub.cfg /tmp/tmp.tXsPznABtK 2023-12-04T05:18:47.836 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.tXsPznABtK 2023-12-04T05:18:47.949 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.tXsPznABtK is 10KB 2023-12-04T05:18:47.962 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.tXsPznABtK 2023-12-04T05:18:47.969 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-12-04T05:18:48.020 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:18:48.020 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-12-04T05:18:48.020 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-12-04T05:18:48.083 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-89-generic 2023-12-04T05:18:48.083 DEBUG:teuthology.orchestra.run.smithi107:> sudo update-grub 2023-12-04T05:18:49.443 INFO:teuthology.orchestra.run.smithi107.stderr:Sourcing file `/etc/default/grub' 2023-12-04T05:18:49.455 INFO:teuthology.orchestra.run.smithi107.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-04T05:18:49.459 INFO:teuthology.orchestra.run.smithi107.stderr:Generating grub configuration file ... 2023-12-04T05:18:49.658 INFO:teuthology.orchestra.run.smithi107.stderr:Found linux image: /boot/vmlinuz-5.15.0-89-generic 2023-12-04T05:18:49.663 INFO:teuthology.orchestra.run.smithi107.stderr:Found initrd image: /boot/initrd.img-5.15.0-89-generic 2023-12-04T05:18:49.934 INFO:teuthology.orchestra.run.smithi107.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-12-04T05:18:49.940 INFO:teuthology.orchestra.run.smithi107.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-12-04T05:18:49.983 INFO:teuthology.orchestra.run.smithi107.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-12-04T05:18:49.988 INFO:teuthology.orchestra.run.smithi107.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-12-04T05:18:50.114 INFO:teuthology.orchestra.run.smithi107.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-12-04T05:18:50.114 INFO:teuthology.orchestra.run.smithi107.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-12-04T05:18:50.114 INFO:teuthology.orchestra.run.smithi107.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-12-04T05:18:50.125 INFO:teuthology.orchestra.run.smithi107.stderr:done 2023-12-04T05:18:50.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo shutdown -r now 2023-12-04T05:18:51.319 INFO:teuthology.orchestra.run.smithi195.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-12-04T05:18:51.319 INFO:teuthology.orchestra.run.smithi195.stdout:Sourcing file `/etc/default/grub' 2023-12-04T05:18:51.327 INFO:teuthology.orchestra.run.smithi195.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-04T05:18:51.343 INFO:teuthology.orchestra.run.smithi195.stdout:Generating grub configuration file ... 2023-12-04T05:18:51.643 INFO:teuthology.orchestra.run.smithi195.stdout:Found linux image: /boot/vmlinuz-5.15.0-89-generic 2023-12-04T05:18:51.670 INFO:teuthology.orchestra.run.smithi195.stdout:Found initrd image: /boot/initrd.img-5.15.0-89-generic 2023-12-04T05:18:51.954 INFO:teuthology.orchestra.run.smithi195.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-12-04T05:18:51.960 INFO:teuthology.orchestra.run.smithi195.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-12-04T05:18:52.024 INFO:teuthology.orchestra.run.smithi195.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-12-04T05:18:52.030 INFO:teuthology.orchestra.run.smithi195.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-12-04T05:18:52.204 INFO:teuthology.orchestra.run.smithi195.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-12-04T05:18:52.204 INFO:teuthology.orchestra.run.smithi195.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-12-04T05:18:52.204 INFO:teuthology.orchestra.run.smithi195.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-12-04T05:18:52.247 INFO:teuthology.orchestra.run.smithi195.stdout:done 2023-12-04T05:18:52.737 DEBUG:teuthology.orchestra.run.smithi195:> dpkg -s linux-image-generic 2023-12-04T05:18:52.762 INFO:teuthology.orchestra.run.smithi195.stdout:Package: linux-image-generic 2023-12-04T05:18:52.762 INFO:teuthology.orchestra.run.smithi195.stdout:Status: install ok installed 2023-12-04T05:18:52.762 INFO:teuthology.orchestra.run.smithi195.stdout:Priority: optional 2023-12-04T05:18:52.762 INFO:teuthology.orchestra.run.smithi195.stdout:Section: kernel 2023-12-04T05:18:52.762 INFO:teuthology.orchestra.run.smithi195.stdout:Installed-Size: 20 2023-12-04T05:18:52.762 INFO:teuthology.orchestra.run.smithi195.stdout:Maintainer: Ubuntu Kernel Team 2023-12-04T05:18:52.763 INFO:teuthology.orchestra.run.smithi195.stdout:Architecture: amd64 2023-12-04T05:18:52.763 INFO:teuthology.orchestra.run.smithi195.stdout:Source: linux-meta 2023-12-04T05:18:52.763 INFO:teuthology.orchestra.run.smithi195.stdout:Version: 5.15.0.89.86 2023-12-04T05:18:52.763 INFO:teuthology.orchestra.run.smithi195.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-89), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-12-04T05:18:52.763 INFO:teuthology.orchestra.run.smithi195.stdout:Depends: linux-image-5.15.0-89-generic, linux-modules-extra-5.15.0-89-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-04T05:18:52.763 INFO:teuthology.orchestra.run.smithi195.stdout:Recommends: thermald 2023-12-04T05:18:52.763 INFO:teuthology.orchestra.run.smithi195.stdout:Description: Generic Linux kernel image 2023-12-04T05:18:52.763 INFO:teuthology.orchestra.run.smithi195.stdout: This package will always depend on the latest generic kernel image 2023-12-04T05:18:52.763 INFO:teuthology.orchestra.run.smithi195.stdout: available. 2023-12-04T05:18:52.763 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-89-generic 2023-12-04T05:18:52.763 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T05:18:52.763 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T05:18:52.764 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-04T05:18:52.764 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi195.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T05:18:52.764 DEBUG:teuthology.orchestra.run.smithi195:> sudo apt-get clean 2023-12-04T05:18:52.933 DEBUG:teuthology.orchestra.run.smithi195:> sudo apt-get update 2023-12-04T05:18:53.060 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-04T05:18:53.069 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-12-04T05:18:53.090 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-12-04T05:18:53.195 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-12-04T05:18:54.258 INFO:teuthology.orchestra.run.smithi195.stdout:Reading package lists... 2023-12-04T05:18:54.276 DEBUG:teuthology.orchestra.run.smithi195:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-04T05:18:54.327 INFO:teuthology.orchestra.run.smithi195.stdout:Reading package lists... 2023-12-04T05:18:54.495 INFO:teuthology.orchestra.run.smithi195.stdout:Building dependency tree... 2023-12-04T05:18:54.495 INFO:teuthology.orchestra.run.smithi195.stdout:Reading state information... 2023-12-04T05:18:54.641 INFO:teuthology.orchestra.run.smithi195.stdout:linux-image-generic is already the newest version (5.15.0.89.86). 2023-12-04T05:18:54.642 INFO:teuthology.orchestra.run.smithi195.stdout:The following packages were automatically installed and are no longer required: 2023-12-04T05:18:54.642 INFO:teuthology.orchestra.run.smithi195.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-04T05:18:54.642 INFO:teuthology.orchestra.run.smithi195.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-04T05:18:54.642 INFO:teuthology.orchestra.run.smithi195.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-04T05:18:54.642 INFO:teuthology.orchestra.run.smithi195.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-04T05:18:54.642 INFO:teuthology.orchestra.run.smithi195.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-04T05:18:54.642 INFO:teuthology.orchestra.run.smithi195.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-04T05:18:54.677 INFO:teuthology.orchestra.run.smithi195.stdout:0 upgraded, 0 newly installed, 0 to remove and 274 not upgraded. 2023-12-04T05:18:54.679 DEBUG:teuthology.orchestra.run.smithi195:> dpkg -s linux-image-generic 2023-12-04T05:18:54.732 INFO:teuthology.orchestra.run.smithi195.stdout:Package: linux-image-generic 2023-12-04T05:18:54.733 INFO:teuthology.orchestra.run.smithi195.stdout:Status: install ok installed 2023-12-04T05:18:54.733 INFO:teuthology.orchestra.run.smithi195.stdout:Priority: optional 2023-12-04T05:18:54.733 INFO:teuthology.orchestra.run.smithi195.stdout:Section: kernel 2023-12-04T05:18:54.733 INFO:teuthology.orchestra.run.smithi195.stdout:Installed-Size: 20 2023-12-04T05:18:54.733 INFO:teuthology.orchestra.run.smithi195.stdout:Maintainer: Ubuntu Kernel Team 2023-12-04T05:18:54.733 INFO:teuthology.orchestra.run.smithi195.stdout:Architecture: amd64 2023-12-04T05:18:54.733 INFO:teuthology.orchestra.run.smithi195.stdout:Source: linux-meta 2023-12-04T05:18:54.733 INFO:teuthology.orchestra.run.smithi195.stdout:Version: 5.15.0.89.86 2023-12-04T05:18:54.733 INFO:teuthology.orchestra.run.smithi195.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-89), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-12-04T05:18:54.733 INFO:teuthology.orchestra.run.smithi195.stdout:Depends: linux-image-5.15.0-89-generic, linux-modules-extra-5.15.0-89-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-04T05:18:54.733 INFO:teuthology.orchestra.run.smithi195.stdout:Recommends: thermald 2023-12-04T05:18:54.733 INFO:teuthology.orchestra.run.smithi195.stdout:Description: Generic Linux kernel image 2023-12-04T05:18:54.733 INFO:teuthology.orchestra.run.smithi195.stdout: This package will always depend on the latest generic kernel image 2023-12-04T05:18:54.734 INFO:teuthology.orchestra.run.smithi195.stdout: available. 2023-12-04T05:18:54.734 DEBUG:teuthology.orchestra.run.smithi195:> mktemp 2023-12-04T05:18:54.778 INFO:teuthology.orchestra.run.smithi195.stdout:/tmp/tmp.p6RkN1DE5t 2023-12-04T05:18:54.779 DEBUG:teuthology.orchestra.run.smithi195:> sudo cp /boot/grub/grub.cfg /tmp/tmp.p6RkN1DE5t 2023-12-04T05:18:54.833 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0666 /tmp/tmp.p6RkN1DE5t 2023-12-04T05:18:54.956 DEBUG:teuthology.orchestra.remote:smithi195:/tmp/tmp.p6RkN1DE5t is 10KB 2023-12-04T05:18:54.968 DEBUG:teuthology.orchestra.run.smithi195:> rm -fr /tmp/tmp.p6RkN1DE5t 2023-12-04T05:18:54.974 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-12-04T05:18:55.028 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:18:55.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-12-04T05:18:55.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-12-04T05:18:55.093 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-89-generic 2023-12-04T05:18:55.093 DEBUG:teuthology.orchestra.run.smithi195:> sudo update-grub 2023-12-04T05:18:56.166 INFO:teuthology.orchestra.run.smithi195.stderr:Sourcing file `/etc/default/grub' 2023-12-04T05:18:56.167 INFO:teuthology.orchestra.run.smithi195.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-04T05:18:56.169 INFO:teuthology.orchestra.run.smithi195.stderr:Generating grub configuration file ... 2023-12-04T05:18:56.363 INFO:teuthology.orchestra.run.smithi195.stderr:Found linux image: /boot/vmlinuz-5.15.0-89-generic 2023-12-04T05:18:56.369 INFO:teuthology.orchestra.run.smithi195.stderr:Found initrd image: /boot/initrd.img-5.15.0-89-generic 2023-12-04T05:18:56.599 INFO:teuthology.orchestra.run.smithi195.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-12-04T05:18:56.604 INFO:teuthology.orchestra.run.smithi195.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-12-04T05:18:56.643 INFO:teuthology.orchestra.run.smithi195.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-12-04T05:18:56.647 INFO:teuthology.orchestra.run.smithi195.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-12-04T05:18:56.747 INFO:teuthology.orchestra.run.smithi195.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-12-04T05:18:56.748 INFO:teuthology.orchestra.run.smithi195.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-12-04T05:18:56.748 INFO:teuthology.orchestra.run.smithi195.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-12-04T05:18:56.768 INFO:teuthology.orchestra.run.smithi195.stderr:done 2023-12-04T05:18:56.784 DEBUG:teuthology.orchestra.run.smithi195:> sudo shutdown -r now 2023-12-04T05:19:20.167 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-04T05:19:20.168 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-12-04T05:19:20.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T05:19:26.838 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-04T05:19:26.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2023-12-04T05:19:26.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T05:19:38.598 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-12-04T05:19:47.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-12-04T05:19:47.608 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T05:19:50.658 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-12-04T05:20:02.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-12-04T05:20:02.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T05:20:21.095 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-12-04T05:20:26.846 DEBUG:teuthology.orchestra.remote:timed out 2023-12-04T05:20:35.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2023-12-04T05:20:35.856 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T05:20:36.096 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-12-04T05:20:36.097 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T05:20:36.098 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-12-04T05:20:36.603 DEBUG:teuthology.orchestra.run.smithi195:> true 2023-12-04T05:20:37.330 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2023-12-04T05:20:37.330 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-89-generic"... 2023-12-04T05:20:37.330 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2023-12-04T05:20:37.376 INFO:teuthology.orchestra.run.smithi195.stdout:5.15.0-89-generic 2023-12-04T05:20:37.377 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-89-generic vs 5.15.0-89-generic 2023-12-04T05:20:37.377 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T05:20:37.377 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-12-04T05:20:38.378 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-04T05:20:38.378 DEBUG:teuthology.orchestra.run.smithi195:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T05:20:38.462 INFO:teuthology.orchestra.run.smithi195.stdout:ttyS1 2023-12-04T05:20:38.483 DEBUG:teuthology.parallel:result is None 2023-12-04T05:20:54.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-12-04T05:20:54.100 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T05:20:54.434 DEBUG:teuthology.orchestra.run.smithi107:> true 2023-12-04T05:20:55.046 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-12-04T05:20:55.046 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-89-generic"... 2023-12-04T05:20:55.046 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2023-12-04T05:20:55.094 INFO:teuthology.orchestra.run.smithi107.stdout:5.15.0-89-generic 2023-12-04T05:20:55.094 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-89-generic vs 5.15.0-89-generic 2023-12-04T05:20:55.094 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T05:20:55.094 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-12-04T05:20:56.095 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-04T05:20:56.096 DEBUG:teuthology.orchestra.run.smithi107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T05:20:56.442 INFO:teuthology.orchestra.run.smithi107.stdout:ttyS1 2023-12-04T05:20:56.478 DEBUG:teuthology.parallel:result is None 2023-12-04T05:20:56.479 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-04T05:20:56.492 INFO:teuthology.task.internal:Creating test directory... 2023-12-04T05:20:56.492 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T05:20:56.501 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T05:20:56.508 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-04T05:20:56.540 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-04T05:20:56.557 INFO:teuthology.task.internal:Creating archive directory... 2023-12-04T05:20:56.557 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T05:20:56.559 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T05:20:56.658 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-04T05:20:56.669 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-04T05:20:56.670 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T05:20:56.672 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T05:20:56.692 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T05:20:56.695 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T05:20:56.700 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T05:20:56.705 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T05:20:56.707 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-04T05:20:56.719 INFO:teuthology.task.internal:Configuring sudo... 2023-12-04T05:20:56.719 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T05:20:56.746 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T05:20:56.775 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-04T05:20:56.788 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-04T05:20:56.788 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T05:20:56.802 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T05:20:56.821 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T05:20:56.856 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T05:20:56.906 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:20:56.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T05:20:56.999 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T05:20:57.011 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T05:20:57.057 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:20:57.057 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T05:20:57.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2023-12-04T05:20:57.116 DEBUG:teuthology.orchestra.run.smithi195:> sudo service rsyslog restart 2023-12-04T05:20:57.209 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-04T05:20:57.220 INFO:teuthology.task.internal:Starting timer... 2023-12-04T05:20:57.220 INFO:teuthology.run_tasks:Running task pcp... 2023-12-04T05:20:57.237 INFO:teuthology.run_tasks:Running task selinux... 2023-12-04T05:20:57.252 DEBUG:teuthology.task.selinux:Excluding smithi107: OS 'ubuntu' does not support SELinux 2023-12-04T05:20:57.252 DEBUG:teuthology.task.selinux:Excluding smithi195: OS 'ubuntu' does not support SELinux 2023-12-04T05:20:57.254 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-04T05:20:57.254 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-12-04T05:20:57.254 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-04T05:20:57.254 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-04T05:20:57.272 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-04T05:20:57.286 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-04T05:20:57.534 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-04T05:20:57.574 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-04T05:20:57.575 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi107.front.sepia.ceph.com,smithi195.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-04T05:36:10.472 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi107.front.sepia.ceph.com'), Remote(name='ubuntu@smithi195.front.sepia.ceph.com')] 2023-12-04T05:36:10.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-12-04T05:36:10.474 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T05:36:10.550 DEBUG:teuthology.orchestra.run.smithi107:> true 2023-12-04T05:36:10.658 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-12-04T05:36:10.659 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2023-12-04T05:36:10.659 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T05:36:10.735 DEBUG:teuthology.orchestra.run.smithi195:> true 2023-12-04T05:36:10.838 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2023-12-04T05:36:10.838 INFO:teuthology.run_tasks:Running task clock... 2023-12-04T05:36:10.850 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-04T05:36:10.850 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-04T05:36:10.851 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-04T05:36:10.853 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-04T05:36:10.853 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-04T05:36:10.881 INFO:teuthology.orchestra.run.smithi107.stdout: 4 Dec 05:36:10 ntpd[28035]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-12-04T05:36:10.881 INFO:teuthology.orchestra.run.smithi107.stdout: 4 Dec 05:36:10 ntpd[28035]: Command line: ntpd -gq 2023-12-04T05:36:10.881 INFO:teuthology.orchestra.run.smithi107.stdout: 4 Dec 05:36:10 ntpd[28035]: ---------------------------------------------------- 2023-12-04T05:36:10.881 INFO:teuthology.orchestra.run.smithi107.stdout: 4 Dec 05:36:10 ntpd[28035]: ntp-4 is maintained by Network Time Foundation, 2023-12-04T05:36:10.882 INFO:teuthology.orchestra.run.smithi107.stdout: 4 Dec 05:36:10 ntpd[28035]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-12-04T05:36:10.882 INFO:teuthology.orchestra.run.smithi107.stdout: 4 Dec 05:36:10 ntpd[28035]: corporation. Support and training for ntp-4 are 2023-12-04T05:36:10.882 INFO:teuthology.orchestra.run.smithi107.stdout: 4 Dec 05:36:10 ntpd[28035]: available at https://www.nwtime.org/support 2023-12-04T05:36:10.882 INFO:teuthology.orchestra.run.smithi107.stdout: 4 Dec 05:36:10 ntpd[28035]: ---------------------------------------------------- 2023-12-04T05:36:10.883 INFO:teuthology.orchestra.run.smithi107.stdout: 4 Dec 05:36:10 ntpd[28035]: proto: precision = 0.046 usec (-24) 2023-12-04T05:36:10.883 INFO:teuthology.orchestra.run.smithi107.stdout: 4 Dec 05:36:10 ntpd[28035]: basedate set to 2022-02-04 2023-12-04T05:36:10.883 INFO:teuthology.orchestra.run.smithi107.stdout: 4 Dec 05:36:10 ntpd[28035]: gps base set to 2022-02-06 (week 2196) 2023-12-04T05:36:10.883 INFO:teuthology.orchestra.run.smithi107.stdout: 4 Dec 05:36:10 ntpd[28035]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-04T05:36:10.883 INFO:teuthology.orchestra.run.smithi107.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-04T05:36:10.883 INFO:teuthology.orchestra.run.smithi107.stderr:restrict ::: KOD does nothing without LIMITED. 2023-12-04T05:36:10.888 INFO:teuthology.orchestra.run.smithi107.stdout: 4 Dec 05:36:10 ntpd[28035]: restrict ::: KOD does nothing without LIMITED. 2023-12-04T05:36:10.888 INFO:teuthology.orchestra.run.smithi107.stdout: 4 Dec 05:36:10 ntpd[28035]: Listen and drop on 0 v6wildcard [::]:123 2023-12-04T05:36:10.889 INFO:teuthology.orchestra.run.smithi107.stdout: 4 Dec 05:36:10 ntpd[28035]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-12-04T05:36:10.889 INFO:teuthology.orchestra.run.smithi107.stdout: 4 Dec 05:36:10 ntpd[28035]: Listen normally on 2 lo 127.0.0.1:123 2023-12-04T05:36:10.889 INFO:teuthology.orchestra.run.smithi107.stdout: 4 Dec 05:36:10 ntpd[28035]: Listen normally on 3 enp3s0f1 172.21.15.107:123 2023-12-04T05:36:10.890 INFO:teuthology.orchestra.run.smithi107.stdout: 4 Dec 05:36:10 ntpd[28035]: Listen normally on 4 lo [::1]:123 2023-12-04T05:36:10.890 INFO:teuthology.orchestra.run.smithi107.stdout: 4 Dec 05:36:10 ntpd[28035]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:47eb%5]:123 2023-12-04T05:36:10.890 INFO:teuthology.orchestra.run.smithi107.stdout: 4 Dec 05:36:10 ntpd[28035]: Listening on routing socket on fd #22 for interface updates 2023-12-04T05:36:10.906 INFO:teuthology.orchestra.run.smithi195.stdout: 4 Dec 05:36:10 ntpd[28028]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-12-04T05:36:10.906 INFO:teuthology.orchestra.run.smithi195.stdout: 4 Dec 05:36:10 ntpd[28028]: Command line: ntpd -gq 2023-12-04T05:36:10.906 INFO:teuthology.orchestra.run.smithi195.stdout: 4 Dec 05:36:10 ntpd[28028]: ---------------------------------------------------- 2023-12-04T05:36:10.906 INFO:teuthology.orchestra.run.smithi195.stdout: 4 Dec 05:36:10 ntpd[28028]: ntp-4 is maintained by Network Time Foundation, 2023-12-04T05:36:10.906 INFO:teuthology.orchestra.run.smithi195.stdout: 4 Dec 05:36:10 ntpd[28028]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-12-04T05:36:10.906 INFO:teuthology.orchestra.run.smithi195.stdout: 4 Dec 05:36:10 ntpd[28028]: corporation. Support and training for ntp-4 are 2023-12-04T05:36:10.906 INFO:teuthology.orchestra.run.smithi195.stdout: 4 Dec 05:36:10 ntpd[28028]: available at https://www.nwtime.org/support 2023-12-04T05:36:10.906 INFO:teuthology.orchestra.run.smithi195.stdout: 4 Dec 05:36:10 ntpd[28028]: ---------------------------------------------------- 2023-12-04T05:36:10.907 INFO:teuthology.orchestra.run.smithi195.stdout: 4 Dec 05:36:10 ntpd[28028]: proto: precision = 0.054 usec (-24) 2023-12-04T05:36:10.907 INFO:teuthology.orchestra.run.smithi195.stdout: 4 Dec 05:36:10 ntpd[28028]: basedate set to 2022-02-04 2023-12-04T05:36:10.907 INFO:teuthology.orchestra.run.smithi195.stdout: 4 Dec 05:36:10 ntpd[28028]: gps base set to 2022-02-06 (week 2196) 2023-12-04T05:36:10.907 INFO:teuthology.orchestra.run.smithi195.stdout: 4 Dec 05:36:10 ntpd[28028]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-04T05:36:10.907 INFO:teuthology.orchestra.run.smithi195.stdout: 4 Dec 05:36:10 ntpd[28028]: restrict ::: KOD does nothing without LIMITED. 2023-12-04T05:36:10.908 INFO:teuthology.orchestra.run.smithi195.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-04T05:36:10.908 INFO:teuthology.orchestra.run.smithi195.stderr:restrict ::: KOD does nothing without LIMITED. 2023-12-04T05:36:10.908 INFO:teuthology.orchestra.run.smithi195.stdout: 4 Dec 05:36:10 ntpd[28028]: Listen and drop on 0 v6wildcard [::]:123 2023-12-04T05:36:10.909 INFO:teuthology.orchestra.run.smithi195.stdout: 4 Dec 05:36:10 ntpd[28028]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-12-04T05:36:10.909 INFO:teuthology.orchestra.run.smithi195.stdout: 4 Dec 05:36:10 ntpd[28028]: Listen normally on 2 lo 127.0.0.1:123 2023-12-04T05:36:10.909 INFO:teuthology.orchestra.run.smithi195.stdout: 4 Dec 05:36:10 ntpd[28028]: Listen normally on 3 enp3s0f1 172.21.15.195:123 2023-12-04T05:36:10.909 INFO:teuthology.orchestra.run.smithi195.stdout: 4 Dec 05:36:10 ntpd[28028]: Listen normally on 4 lo [::1]:123 2023-12-04T05:36:10.909 INFO:teuthology.orchestra.run.smithi195.stdout: 4 Dec 05:36:10 ntpd[28028]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:8081%5]:123 2023-12-04T05:36:10.909 INFO:teuthology.orchestra.run.smithi195.stdout: 4 Dec 05:36:10 ntpd[28028]: Listening on routing socket on fd #22 for interface updates 2023-12-04T05:36:11.884 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:11 ntpd[28035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:11.884 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:11 ntpd[28035]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:11.906 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:11 ntpd[28028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:11.907 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:11 ntpd[28028]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:11.907 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:11 ntpd[28028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:11.907 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:11 ntpd[28028]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:11.907 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:11 ntpd[28028]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T05:36:11.907 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:11 ntpd[28028]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T05:36:13.882 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:13 ntpd[28035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:13.882 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:13 ntpd[28035]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:13.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:13 ntpd[28035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:13.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:13 ntpd[28035]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:13.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:13 ntpd[28035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T05:36:13.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:13 ntpd[28035]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T05:36:13.907 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:13 ntpd[28028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:13.907 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:13 ntpd[28028]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:13.907 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:13 ntpd[28028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:13.907 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:13 ntpd[28028]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:13.907 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:13 ntpd[28028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:13.907 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:13 ntpd[28028]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:13.907 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:13 ntpd[28028]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T05:36:13.907 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:13 ntpd[28028]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T05:36:13.907 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:13 ntpd[28028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:13.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:13 ntpd[28028]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:13.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:13 ntpd[28028]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T05:36:13.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:13 ntpd[28028]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T05:36:14.882 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:14 ntpd[28035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:14.882 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:14 ntpd[28035]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:14.882 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:14 ntpd[28035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:14.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:14 ntpd[28035]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:14.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:14 ntpd[28035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:14.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:14 ntpd[28035]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:14.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:14 ntpd[28035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T05:36:14.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:14 ntpd[28035]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T05:36:14.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:14 ntpd[28035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:14.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:14 ntpd[28035]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:14.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:14 ntpd[28035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T05:36:14.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:14 ntpd[28035]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T05:36:14.906 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:14 ntpd[28028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:14.907 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:14 ntpd[28028]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:14.907 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:14 ntpd[28028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:14.907 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:14 ntpd[28028]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:14.907 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:14 ntpd[28028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:14.907 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:14 ntpd[28028]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:14.907 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:14 ntpd[28028]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T05:36:14.907 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:14 ntpd[28028]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T05:36:15.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:15 ntpd[28035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:15.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:15 ntpd[28035]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:15.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:15 ntpd[28035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:15.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:15 ntpd[28035]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:15.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:15 ntpd[28035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T05:36:15.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:15 ntpd[28035]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T05:36:15.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:15 ntpd[28028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:15.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:15 ntpd[28028]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:15.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:15 ntpd[28028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:15.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:15 ntpd[28028]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:15.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:15 ntpd[28028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:15.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:15 ntpd[28028]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:15.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:15 ntpd[28028]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T05:36:15.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:15 ntpd[28028]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T05:36:15.909 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:15 ntpd[28028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:15.909 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:15 ntpd[28028]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:15.909 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:15 ntpd[28028]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T05:36:15.909 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:15 ntpd[28028]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T05:36:16.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:16 ntpd[28035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:16.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:16 ntpd[28035]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:16.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:16 ntpd[28035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:16.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:16 ntpd[28035]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:16.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:16 ntpd[28035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:16.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:16 ntpd[28035]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:16.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:16 ntpd[28035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T05:36:16.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:16 ntpd[28035]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T05:36:16.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:16 ntpd[28035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:16.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:16 ntpd[28035]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:16.884 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:16 ntpd[28035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T05:36:16.884 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:16 ntpd[28035]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T05:36:16.910 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:16 ntpd[28028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:16.910 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:16 ntpd[28028]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:16.910 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:16 ntpd[28028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:16.910 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:16 ntpd[28028]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:16.910 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:16 ntpd[28028]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T05:36:16.910 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:16 ntpd[28028]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T05:36:17.882 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:17 ntpd[28035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:17.882 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:17 ntpd[28035]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:17.882 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:17 ntpd[28035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:17.882 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:17 ntpd[28035]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:17.882 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:17 ntpd[28035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T05:36:17.882 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:17 ntpd[28035]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T05:36:17.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:17 ntpd[28028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:17.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:17 ntpd[28028]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:17.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:17 ntpd[28028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:17.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:17 ntpd[28028]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:17.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:17 ntpd[28028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:17.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:17 ntpd[28028]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:17.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:17 ntpd[28028]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T05:36:17.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:17 ntpd[28028]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T05:36:17.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:17 ntpd[28028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:17.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:17 ntpd[28028]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:17.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:17 ntpd[28028]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T05:36:17.908 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:17 ntpd[28028]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T05:36:17.909 INFO:teuthology.orchestra.run.smithi195.stdout: 4 Dec 05:36:17 ntpd[28028]: ntpd: time slew -0.005090 s 2023-12-04T05:36:17.909 INFO:teuthology.orchestra.run.smithi195.stdout:ntpd: time slew -0.005090s 2023-12-04T05:36:17.909 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:17 ntpd[28028]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-12-04T05:36:17.909 INFO:teuthology.orchestra.run.smithi195.stderr: 4 Dec 05:36:17 ntpd[28028]: can't open /var/log/ntpstats/loopstats.20231204: Permission denied 2023-12-04T05:36:18.010 INFO:teuthology.orchestra.run.smithi195.stdout: remote refid st t when poll reach delay offset jitter 2023-12-04T05:36:18.010 INFO:teuthology.orchestra.run.smithi195.stdout:============================================================================== 2023-12-04T05:36:18.010 INFO:teuthology.orchestra.run.smithi195.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-04T05:36:18.010 INFO:teuthology.orchestra.run.smithi195.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-04T05:36:18.010 INFO:teuthology.orchestra.run.smithi195.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-04T05:36:18.010 INFO:teuthology.orchestra.run.smithi195.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-04T05:36:18.882 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:18 ntpd[28035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:18.882 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:18 ntpd[28035]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:18.882 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:18 ntpd[28035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:18.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:18 ntpd[28035]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:18.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:18 ntpd[28035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:18.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:18 ntpd[28035]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:18.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:18 ntpd[28035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T05:36:18.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:18 ntpd[28035]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T05:36:18.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:18 ntpd[28035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:18.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:18 ntpd[28035]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:18.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:18 ntpd[28035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T05:36:18.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:18 ntpd[28035]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T05:36:19.882 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:19 ntpd[28035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:19.882 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:19 ntpd[28035]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:19.882 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:19 ntpd[28035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T05:36:19.882 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:19 ntpd[28035]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T05:36:19.882 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:19 ntpd[28035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T05:36:19.882 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:19 ntpd[28035]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T05:36:19.882 INFO:teuthology.orchestra.run.smithi107.stdout: 4 Dec 05:36:19 ntpd[28035]: ntpd: time slew -0.003150 s 2023-12-04T05:36:19.883 INFO:teuthology.orchestra.run.smithi107.stdout:ntpd: time slew -0.003150s 2023-12-04T05:36:19.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:19 ntpd[28035]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-12-04T05:36:19.883 INFO:teuthology.orchestra.run.smithi107.stderr: 4 Dec 05:36:19 ntpd[28035]: can't open /var/log/ntpstats/loopstats.20231204: Permission denied 2023-12-04T05:36:19.974 INFO:teuthology.orchestra.run.smithi107.stdout: remote refid st t when poll reach delay offset jitter 2023-12-04T05:36:19.974 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================== 2023-12-04T05:36:19.974 INFO:teuthology.orchestra.run.smithi107.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-04T05:36:19.974 INFO:teuthology.orchestra.run.smithi107.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-04T05:36:19.974 INFO:teuthology.orchestra.run.smithi107.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-04T05:36:19.974 INFO:teuthology.orchestra.run.smithi107.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-04T05:36:19.975 INFO:teuthology.run_tasks:Running task install... 2023-12-04T05:36:19.987 DEBUG:teuthology.task.install:project ceph 2023-12-04T05:36:19.987 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'}} 2023-12-04T05:36:19.987 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'} 2023-12-04T05:36:19.987 INFO:teuthology.task.install:Using flavor: default 2023-12-04T05:36:19.994 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-04T05:36:19.995 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-12-04T05:36:19.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt-key list | grep Ceph 2023-12-04T05:36:19.997 DEBUG:teuthology.orchestra.run.smithi195:> sudo apt-key list | grep Ceph 2023-12-04T05:36:20.056 INFO:teuthology.orchestra.run.smithi195.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-12-04T05:36:20.070 INFO:teuthology.orchestra.run.smithi107.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-12-04T05:36:20.083 INFO:teuthology.orchestra.run.smithi195.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-12-04T05:36:20.083 INFO:teuthology.orchestra.run.smithi195.stdout:uid [ unknown] Ceph.com (release key) 2023-12-04T05:36:20.083 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-12-04T05:36:20.084 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-12-04T05:36:20.084 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=47c1c547372ed5bff7984647dc200bac0e019caf 2023-12-04T05:36:20.096 INFO:teuthology.orchestra.run.smithi107.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-12-04T05:36:20.097 INFO:teuthology.orchestra.run.smithi107.stdout:uid [ unknown] Ceph.com (release key) 2023-12-04T05:36:20.097 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-12-04T05:36:20.097 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-12-04T05:36:20.097 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=47c1c547372ed5bff7984647dc200bac0e019caf 2023-12-04T05:36:20.246 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default/ 2023-12-04T05:36:20.246 INFO:teuthology.task.install.deb:Package version is 18.0.0-7531-g47c1c547-1jammy 2023-12-04T05:36:20.393 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default/ 2023-12-04T05:36:20.393 INFO:teuthology.task.install.deb:Package version is 18.0.0-7531-g47c1c547-1jammy 2023-12-04T05:36:20.498 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:36:20.499 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-12-04T05:36:20.501 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:36:20.501 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-12-04T05:36:20.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt-get update 2023-12-04T05:36:20.517 DEBUG:teuthology.orchestra.run.smithi195:> sudo apt-get update 2023-12-04T05:36:20.715 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-12-04T05:36:20.732 INFO:teuthology.orchestra.run.smithi107.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy InRelease 2023-12-04T05:36:20.755 INFO:teuthology.orchestra.run.smithi107.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-12-04T05:36:20.777 INFO:teuthology.orchestra.run.smithi107.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy Release.gpg 2023-12-04T05:36:20.796 INFO:teuthology.orchestra.run.smithi107.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-12-04T05:36:20.815 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-04T05:36:20.816 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-12-04T05:36:20.823 INFO:teuthology.orchestra.run.smithi107.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,403 B] 2023-12-04T05:36:20.868 INFO:teuthology.orchestra.run.smithi195.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy InRelease 2023-12-04T05:36:20.889 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-12-04T05:36:20.913 INFO:teuthology.orchestra.run.smithi195.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-12-04T05:36:20.964 INFO:teuthology.orchestra.run.smithi195.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy Release.gpg 2023-12-04T05:36:20.969 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-12-04T05:36:21.016 INFO:teuthology.orchestra.run.smithi195.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,403 B] 2023-12-04T05:36:21.075 INFO:teuthology.orchestra.run.smithi195.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-12-04T05:36:21.174 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-04T05:36:21.187 INFO:teuthology.orchestra.run.smithi195.stdout:Fetched 28.6 kB in 0s (58.0 kB/s) 2023-12-04T05:36:21.191 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-12-04T05:36:21.208 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-12-04T05:36:21.496 INFO:teuthology.orchestra.run.smithi107.stdout:Fetched 28.6 kB in 1s (36.0 kB/s) 2023-12-04T05:36:22.074 INFO:teuthology.orchestra.run.smithi195.stdout:Reading package lists... 2023-12-04T05:36:22.089 DEBUG:teuthology.orchestra.run.smithi195:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-7531-g47c1c547-1jammy cephadm=18.0.0-7531-g47c1c547-1jammy ceph-mds=18.0.0-7531-g47c1c547-1jammy ceph-mgr=18.0.0-7531-g47c1c547-1jammy ceph-common=18.0.0-7531-g47c1c547-1jammy ceph-fuse=18.0.0-7531-g47c1c547-1jammy ceph-test=18.0.0-7531-g47c1c547-1jammy ceph-volume=18.0.0-7531-g47c1c547-1jammy radosgw=18.0.0-7531-g47c1c547-1jammy python3-rados=18.0.0-7531-g47c1c547-1jammy python3-rgw=18.0.0-7531-g47c1c547-1jammy python3-cephfs=18.0.0-7531-g47c1c547-1jammy python3-rbd=18.0.0-7531-g47c1c547-1jammy libcephfs2=18.0.0-7531-g47c1c547-1jammy libcephfs-dev=18.0.0-7531-g47c1c547-1jammy librados2=18.0.0-7531-g47c1c547-1jammy librbd1=18.0.0-7531-g47c1c547-1jammy rbd-fuse=18.0.0-7531-g47c1c547-1jammy python3-cephfs=18.0.0-7531-g47c1c547-1jammy cephfs-shell=18.0.0-7531-g47c1c547-1jammy cephfs-top=18.0.0-7531-g47c1c547-1jammy cephfs-mirror=18.0.0-7531-g47c1c547-1jammy 2023-12-04T05:36:22.150 INFO:teuthology.orchestra.run.smithi195.stdout:Reading package lists... 2023-12-04T05:36:22.337 INFO:teuthology.orchestra.run.smithi195.stdout:Building dependency tree... 2023-12-04T05:36:22.338 INFO:teuthology.orchestra.run.smithi195.stdout:Reading state information... 2023-12-04T05:36:22.414 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2023-12-04T05:36:22.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-7531-g47c1c547-1jammy cephadm=18.0.0-7531-g47c1c547-1jammy ceph-mds=18.0.0-7531-g47c1c547-1jammy ceph-mgr=18.0.0-7531-g47c1c547-1jammy ceph-common=18.0.0-7531-g47c1c547-1jammy ceph-fuse=18.0.0-7531-g47c1c547-1jammy ceph-test=18.0.0-7531-g47c1c547-1jammy ceph-volume=18.0.0-7531-g47c1c547-1jammy radosgw=18.0.0-7531-g47c1c547-1jammy python3-rados=18.0.0-7531-g47c1c547-1jammy python3-rgw=18.0.0-7531-g47c1c547-1jammy python3-cephfs=18.0.0-7531-g47c1c547-1jammy python3-rbd=18.0.0-7531-g47c1c547-1jammy libcephfs2=18.0.0-7531-g47c1c547-1jammy libcephfs-dev=18.0.0-7531-g47c1c547-1jammy librados2=18.0.0-7531-g47c1c547-1jammy librbd1=18.0.0-7531-g47c1c547-1jammy rbd-fuse=18.0.0-7531-g47c1c547-1jammy python3-cephfs=18.0.0-7531-g47c1c547-1jammy cephfs-shell=18.0.0-7531-g47c1c547-1jammy cephfs-top=18.0.0-7531-g47c1c547-1jammy cephfs-mirror=18.0.0-7531-g47c1c547-1jammy 2023-12-04T05:36:22.492 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2023-12-04T05:36:22.509 INFO:teuthology.orchestra.run.smithi195.stdout:The following packages were automatically installed and are no longer required: 2023-12-04T05:36:22.509 INFO:teuthology.orchestra.run.smithi195.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-04T05:36:22.509 INFO:teuthology.orchestra.run.smithi195.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-04T05:36:22.509 INFO:teuthology.orchestra.run.smithi195.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-04T05:36:22.509 INFO:teuthology.orchestra.run.smithi195.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-04T05:36:22.510 INFO:teuthology.orchestra.run.smithi195.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-04T05:36:22.510 INFO:teuthology.orchestra.run.smithi195.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-04T05:36:22.510 INFO:teuthology.orchestra.run.smithi195.stdout:The following additional packages will be installed: 2023-12-04T05:36:22.510 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-12-04T05:36:22.510 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-12-04T05:36:22.516 INFO:teuthology.orchestra.run.smithi195.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2023-12-04T05:36:22.516 INFO:teuthology.orchestra.run.smithi195.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-12-04T05:36:22.516 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2023-12-04T05:36:22.516 INFO:teuthology.orchestra.run.smithi195.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2023-12-04T05:36:22.516 INFO:teuthology.orchestra.run.smithi195.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-12-04T05:36:22.516 INFO:teuthology.orchestra.run.smithi195.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-12-04T05:36:22.516 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-12-04T05:36:22.516 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-12-04T05:36:22.517 INFO:teuthology.orchestra.run.smithi195.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-12-04T05:36:22.517 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-12-04T05:36:22.517 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-12-04T05:36:22.517 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-12-04T05:36:22.517 INFO:teuthology.orchestra.run.smithi195.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-12-04T05:36:22.517 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-12-04T05:36:22.517 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-12-04T05:36:22.517 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-12-04T05:36:22.517 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-12-04T05:36:22.517 INFO:teuthology.orchestra.run.smithi195.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-12-04T05:36:22.517 INFO:teuthology.orchestra.run.smithi195.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-12-04T05:36:22.517 INFO:teuthology.orchestra.run.smithi195.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-12-04T05:36:22.517 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-12-04T05:36:22.517 INFO:teuthology.orchestra.run.smithi195.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2023-12-04T05:36:22.518 INFO:teuthology.orchestra.run.smithi195.stdout:Suggested packages: 2023-12-04T05:36:22.518 INFO:teuthology.orchestra.run.smithi195.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2023-12-04T05:36:22.518 INFO:teuthology.orchestra.run.smithi195.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2023-12-04T05:36:22.518 INFO:teuthology.orchestra.run.smithi195.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2023-12-04T05:36:22.518 INFO:teuthology.orchestra.run.smithi195.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2023-12-04T05:36:22.518 INFO:teuthology.orchestra.run.smithi195.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2023-12-04T05:36:22.518 INFO:teuthology.orchestra.run.smithi195.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-12-04T05:36:22.518 INFO:teuthology.orchestra.run.smithi195.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2023-12-04T05:36:22.518 INFO:teuthology.orchestra.run.smithi195.stdout:Recommended packages: 2023-12-04T05:36:22.518 INFO:teuthology.orchestra.run.smithi195.stdout: btrfs-tools 2023-12-04T05:36:22.620 INFO:teuthology.orchestra.run.smithi195.stdout:The following NEW packages will be installed: 2023-12-04T05:36:22.620 INFO:teuthology.orchestra.run.smithi195.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-12-04T05:36:22.620 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-12-04T05:36:22.620 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-12-04T05:36:22.621 INFO:teuthology.orchestra.run.smithi195.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-12-04T05:36:22.621 INFO:teuthology.orchestra.run.smithi195.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2023-12-04T05:36:22.621 INFO:teuthology.orchestra.run.smithi195.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-12-04T05:36:22.621 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2023-12-04T05:36:22.621 INFO:teuthology.orchestra.run.smithi195.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2023-12-04T05:36:22.621 INFO:teuthology.orchestra.run.smithi195.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2023-12-04T05:36:22.621 INFO:teuthology.orchestra.run.smithi195.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-12-04T05:36:22.621 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-12-04T05:36:22.621 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-12-04T05:36:22.621 INFO:teuthology.orchestra.run.smithi195.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-12-04T05:36:22.621 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-12-04T05:36:22.621 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-12-04T05:36:22.621 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-12-04T05:36:22.622 INFO:teuthology.orchestra.run.smithi195.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-12-04T05:36:22.622 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-12-04T05:36:22.622 INFO:teuthology.orchestra.run.smithi195.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-12-04T05:36:22.622 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-12-04T05:36:22.622 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-12-04T05:36:22.622 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-12-04T05:36:22.622 INFO:teuthology.orchestra.run.smithi195.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-12-04T05:36:22.622 INFO:teuthology.orchestra.run.smithi195.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-12-04T05:36:22.622 INFO:teuthology.orchestra.run.smithi195.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-12-04T05:36:22.622 INFO:teuthology.orchestra.run.smithi195.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-12-04T05:36:22.622 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-12-04T05:36:22.622 INFO:teuthology.orchestra.run.smithi195.stdout: socat unzip xmlstarlet zip 2023-12-04T05:36:22.675 INFO:teuthology.orchestra.run.smithi107.stdout:Building dependency tree... 2023-12-04T05:36:22.675 INFO:teuthology.orchestra.run.smithi107.stdout:Reading state information... 2023-12-04T05:36:22.806 INFO:teuthology.orchestra.run.smithi195.stdout:0 upgraded, 124 newly installed, 0 to remove and 262 not upgraded. 2023-12-04T05:36:22.806 INFO:teuthology.orchestra.run.smithi195.stdout:Need to get 187 MB of archives. 2023-12-04T05:36:22.806 INFO:teuthology.orchestra.run.smithi195.stdout:After this operation, 835 MB of additional disk space will be used. 2023-12-04T05:36:22.806 INFO:teuthology.orchestra.run.smithi195.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-12-04T05:36:22.848 INFO:teuthology.orchestra.run.smithi107.stdout:The following packages were automatically installed and are no longer required: 2023-12-04T05:36:22.848 INFO:teuthology.orchestra.run.smithi107.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-04T05:36:22.848 INFO:teuthology.orchestra.run.smithi107.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-04T05:36:22.848 INFO:teuthology.orchestra.run.smithi107.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-04T05:36:22.849 INFO:teuthology.orchestra.run.smithi107.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-04T05:36:22.849 INFO:teuthology.orchestra.run.smithi107.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-04T05:36:22.849 INFO:teuthology.orchestra.run.smithi107.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-04T05:36:22.850 INFO:teuthology.orchestra.run.smithi107.stdout:The following additional packages will be installed: 2023-12-04T05:36:22.850 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-12-04T05:36:22.850 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-12-04T05:36:22.850 INFO:teuthology.orchestra.run.smithi107.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2023-12-04T05:36:22.850 INFO:teuthology.orchestra.run.smithi107.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-12-04T05:36:22.852 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2023-12-04T05:36:22.853 INFO:teuthology.orchestra.run.smithi107.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2023-12-04T05:36:22.853 INFO:teuthology.orchestra.run.smithi107.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-12-04T05:36:22.853 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-12-04T05:36:22.853 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-12-04T05:36:22.853 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-12-04T05:36:22.853 INFO:teuthology.orchestra.run.smithi107.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-12-04T05:36:22.853 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-12-04T05:36:22.853 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-12-04T05:36:22.853 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-12-04T05:36:22.853 INFO:teuthology.orchestra.run.smithi107.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-12-04T05:36:22.853 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-12-04T05:36:22.854 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-12-04T05:36:22.854 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-12-04T05:36:22.854 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-12-04T05:36:22.854 INFO:teuthology.orchestra.run.smithi107.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-12-04T05:36:22.854 INFO:teuthology.orchestra.run.smithi107.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-12-04T05:36:22.854 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-12-04T05:36:22.854 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-12-04T05:36:22.854 INFO:teuthology.orchestra.run.smithi107.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2023-12-04T05:36:22.854 INFO:teuthology.orchestra.run.smithi107.stdout:Suggested packages: 2023-12-04T05:36:22.854 INFO:teuthology.orchestra.run.smithi107.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2023-12-04T05:36:22.854 INFO:teuthology.orchestra.run.smithi107.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2023-12-04T05:36:22.855 INFO:teuthology.orchestra.run.smithi107.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2023-12-04T05:36:22.855 INFO:teuthology.orchestra.run.smithi107.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2023-12-04T05:36:22.855 INFO:teuthology.orchestra.run.smithi107.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2023-12-04T05:36:22.855 INFO:teuthology.orchestra.run.smithi107.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-12-04T05:36:22.855 INFO:teuthology.orchestra.run.smithi107.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2023-12-04T05:36:22.855 INFO:teuthology.orchestra.run.smithi107.stdout:Recommended packages: 2023-12-04T05:36:22.855 INFO:teuthology.orchestra.run.smithi107.stdout: btrfs-tools 2023-12-04T05:36:22.877 INFO:teuthology.orchestra.run.smithi195.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-7531-g47c1c547-1jammy [3,466 kB] 2023-12-04T05:36:22.888 INFO:teuthology.orchestra.run.smithi195.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-12-04T05:36:22.934 INFO:teuthology.orchestra.run.smithi195.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-12-04T05:36:22.961 INFO:teuthology.orchestra.run.smithi107.stdout:The following NEW packages will be installed: 2023-12-04T05:36:22.961 INFO:teuthology.orchestra.run.smithi107.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-12-04T05:36:22.961 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-12-04T05:36:22.961 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-12-04T05:36:22.961 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-12-04T05:36:22.961 INFO:teuthology.orchestra.run.smithi107.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2023-12-04T05:36:22.962 INFO:teuthology.orchestra.run.smithi107.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-12-04T05:36:22.962 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2023-12-04T05:36:22.962 INFO:teuthology.orchestra.run.smithi107.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2023-12-04T05:36:22.964 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2023-12-04T05:36:22.964 INFO:teuthology.orchestra.run.smithi107.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-12-04T05:36:22.964 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-12-04T05:36:22.964 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-12-04T05:36:22.964 INFO:teuthology.orchestra.run.smithi107.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-12-04T05:36:22.964 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-12-04T05:36:22.964 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-12-04T05:36:22.964 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-12-04T05:36:22.964 INFO:teuthology.orchestra.run.smithi107.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-12-04T05:36:22.965 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-12-04T05:36:22.965 INFO:teuthology.orchestra.run.smithi107.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-12-04T05:36:22.965 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-12-04T05:36:22.965 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-12-04T05:36:22.965 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-12-04T05:36:22.965 INFO:teuthology.orchestra.run.smithi107.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-12-04T05:36:22.965 INFO:teuthology.orchestra.run.smithi107.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-12-04T05:36:22.965 INFO:teuthology.orchestra.run.smithi107.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-12-04T05:36:22.965 INFO:teuthology.orchestra.run.smithi107.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-12-04T05:36:22.965 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-12-04T05:36:22.965 INFO:teuthology.orchestra.run.smithi107.stdout: socat unzip xmlstarlet zip 2023-12-04T05:36:23.021 INFO:teuthology.orchestra.run.smithi107.stdout:0 upgraded, 124 newly installed, 0 to remove and 262 not upgraded. 2023-12-04T05:36:23.021 INFO:teuthology.orchestra.run.smithi107.stdout:Need to get 187 MB of archives. 2023-12-04T05:36:23.021 INFO:teuthology.orchestra.run.smithi107.stdout:After this operation, 835 MB of additional disk space will be used. 2023-12-04T05:36:23.021 INFO:teuthology.orchestra.run.smithi107.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-12-04T05:36:23.038 INFO:teuthology.orchestra.run.smithi107.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-12-04T05:36:23.049 INFO:teuthology.orchestra.run.smithi107.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-12-04T05:36:23.088 INFO:teuthology.orchestra.run.smithi107.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-12-04T05:36:23.104 INFO:teuthology.orchestra.run.smithi107.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-7531-g47c1c547-1jammy [3,466 kB] 2023-12-04T05:36:23.109 INFO:teuthology.orchestra.run.smithi107.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-12-04T05:36:23.111 INFO:teuthology.orchestra.run.smithi107.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-12-04T05:36:23.123 INFO:teuthology.orchestra.run.smithi107.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-12-04T05:36:23.170 INFO:teuthology.orchestra.run.smithi107.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-12-04T05:36:23.174 INFO:teuthology.orchestra.run.smithi107.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-12-04T05:36:23.185 INFO:teuthology.orchestra.run.smithi107.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-12-04T05:36:23.188 INFO:teuthology.orchestra.run.smithi107.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-12-04T05:36:23.189 INFO:teuthology.orchestra.run.smithi107.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-12-04T05:36:23.191 INFO:teuthology.orchestra.run.smithi195.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-12-04T05:36:23.197 INFO:teuthology.orchestra.run.smithi107.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2023-12-04T05:36:23.199 INFO:teuthology.orchestra.run.smithi107.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2023-12-04T05:36:23.201 INFO:teuthology.orchestra.run.smithi107.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2023-12-04T05:36:23.202 INFO:teuthology.orchestra.run.smithi107.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2023-12-04T05:36:23.203 INFO:teuthology.orchestra.run.smithi107.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2023-12-04T05:36:23.206 INFO:teuthology.orchestra.run.smithi107.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] 2023-12-04T05:36:23.208 INFO:teuthology.orchestra.run.smithi107.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2023-12-04T05:36:23.223 INFO:teuthology.orchestra.run.smithi107.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-12-04T05:36:23.223 INFO:teuthology.orchestra.run.smithi107.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-12-04T05:36:23.223 INFO:teuthology.orchestra.run.smithi107.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-12-04T05:36:23.224 INFO:teuthology.orchestra.run.smithi107.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-12-04T05:36:23.224 INFO:teuthology.orchestra.run.smithi107.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-12-04T05:36:23.224 INFO:teuthology.orchestra.run.smithi107.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-12-04T05:36:23.225 INFO:teuthology.orchestra.run.smithi107.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-12-04T05:36:23.225 INFO:teuthology.orchestra.run.smithi107.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-12-04T05:36:23.240 INFO:teuthology.orchestra.run.smithi107.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-12-04T05:36:23.240 INFO:teuthology.orchestra.run.smithi107.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-12-04T05:36:23.256 INFO:teuthology.orchestra.run.smithi107.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-12-04T05:36:23.257 INFO:teuthology.orchestra.run.smithi107.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-12-04T05:36:23.257 INFO:teuthology.orchestra.run.smithi107.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-12-04T05:36:23.257 INFO:teuthology.orchestra.run.smithi107.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-12-04T05:36:23.258 INFO:teuthology.orchestra.run.smithi107.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-12-04T05:36:23.259 INFO:teuthology.orchestra.run.smithi107.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-12-04T05:36:23.259 INFO:teuthology.orchestra.run.smithi107.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-12-04T05:36:23.260 INFO:teuthology.orchestra.run.smithi107.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-12-04T05:36:23.262 INFO:teuthology.orchestra.run.smithi195.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-12-04T05:36:23.264 INFO:teuthology.orchestra.run.smithi195.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-12-04T05:36:23.273 INFO:teuthology.orchestra.run.smithi107.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-12-04T05:36:23.273 INFO:teuthology.orchestra.run.smithi107.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-12-04T05:36:23.289 INFO:teuthology.orchestra.run.smithi107.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-12-04T05:36:23.290 INFO:teuthology.orchestra.run.smithi107.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-12-04T05:36:23.290 INFO:teuthology.orchestra.run.smithi107.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-12-04T05:36:23.291 INFO:teuthology.orchestra.run.smithi107.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-12-04T05:36:23.292 INFO:teuthology.orchestra.run.smithi107.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-12-04T05:36:23.294 INFO:teuthology.orchestra.run.smithi107.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-12-04T05:36:23.294 INFO:teuthology.orchestra.run.smithi107.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2023-12-04T05:36:23.296 INFO:teuthology.orchestra.run.smithi107.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-12-04T05:36:23.306 INFO:teuthology.orchestra.run.smithi107.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-12-04T05:36:23.308 INFO:teuthology.orchestra.run.smithi107.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-12-04T05:36:23.323 INFO:teuthology.orchestra.run.smithi107.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-12-04T05:36:23.323 INFO:teuthology.orchestra.run.smithi107.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-12-04T05:36:23.325 INFO:teuthology.orchestra.run.smithi107.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-12-04T05:36:23.352 INFO:teuthology.orchestra.run.smithi107.stdout:Get:54 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-7531-g47c1c547-1jammy [3,111 kB] 2023-12-04T05:36:23.363 INFO:teuthology.orchestra.run.smithi195.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-12-04T05:36:23.376 INFO:teuthology.orchestra.run.smithi107.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-12-04T05:36:23.377 INFO:teuthology.orchestra.run.smithi107.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-12-04T05:36:23.377 INFO:teuthology.orchestra.run.smithi107.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-12-04T05:36:23.378 INFO:teuthology.orchestra.run.smithi107.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-12-04T05:36:23.379 INFO:teuthology.orchestra.run.smithi107.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-12-04T05:36:23.396 INFO:teuthology.orchestra.run.smithi107.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-12-04T05:36:23.398 INFO:teuthology.orchestra.run.smithi107.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-12-04T05:36:23.398 INFO:teuthology.orchestra.run.smithi107.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-12-04T05:36:23.413 INFO:teuthology.orchestra.run.smithi107.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-12-04T05:36:23.413 INFO:teuthology.orchestra.run.smithi107.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-12-04T05:36:23.413 INFO:teuthology.orchestra.run.smithi107.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-12-04T05:36:23.413 INFO:teuthology.orchestra.run.smithi107.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-12-04T05:36:23.414 INFO:teuthology.orchestra.run.smithi107.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-12-04T05:36:23.414 INFO:teuthology.orchestra.run.smithi107.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-12-04T05:36:23.428 INFO:teuthology.orchestra.run.smithi107.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-12-04T05:36:23.429 INFO:teuthology.orchestra.run.smithi107.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-12-04T05:36:23.433 INFO:teuthology.orchestra.run.smithi107.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-12-04T05:36:23.446 INFO:teuthology.orchestra.run.smithi107.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-12-04T05:36:23.446 INFO:teuthology.orchestra.run.smithi107.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-12-04T05:36:23.446 INFO:teuthology.orchestra.run.smithi107.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-12-04T05:36:23.449 INFO:teuthology.orchestra.run.smithi107.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-12-04T05:36:23.452 INFO:teuthology.orchestra.run.smithi195.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-7531-g47c1c547-1jammy [3,111 kB] 2023-12-04T05:36:23.452 INFO:teuthology.orchestra.run.smithi107.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-12-04T05:36:23.452 INFO:teuthology.orchestra.run.smithi107.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2023-12-04T05:36:23.452 INFO:teuthology.orchestra.run.smithi107.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2023-12-04T05:36:23.461 INFO:teuthology.orchestra.run.smithi107.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-12-04T05:36:23.465 INFO:teuthology.orchestra.run.smithi107.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2023-12-04T05:36:23.465 INFO:teuthology.orchestra.run.smithi107.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-12-04T05:36:23.469 INFO:teuthology.orchestra.run.smithi107.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-7531-g47c1c547-1jammy [690 kB] 2023-12-04T05:36:23.478 INFO:teuthology.orchestra.run.smithi107.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-12-04T05:36:23.478 INFO:teuthology.orchestra.run.smithi107.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-12-04T05:36:23.478 INFO:teuthology.orchestra.run.smithi107.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-12-04T05:36:23.479 INFO:teuthology.orchestra.run.smithi107.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-12-04T05:36:23.480 INFO:teuthology.orchestra.run.smithi107.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-12-04T05:36:23.481 INFO:teuthology.orchestra.run.smithi107.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-12-04T05:36:23.487 INFO:teuthology.orchestra.run.smithi107.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-12-04T05:36:23.494 INFO:teuthology.orchestra.run.smithi195.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-12-04T05:36:23.494 INFO:teuthology.orchestra.run.smithi107.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-12-04T05:36:23.495 INFO:teuthology.orchestra.run.smithi107.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-12-04T05:36:23.496 INFO:teuthology.orchestra.run.smithi107.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-12-04T05:36:23.503 INFO:teuthology.orchestra.run.smithi195.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-12-04T05:36:23.505 INFO:teuthology.orchestra.run.smithi107.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-7531-g47c1c547-1jammy [356 kB] 2023-12-04T05:36:23.511 INFO:teuthology.orchestra.run.smithi107.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-12-04T05:36:23.521 INFO:teuthology.orchestra.run.smithi107.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-7531-g47c1c547-1jammy [32.8 kB] 2023-12-04T05:36:23.522 INFO:teuthology.orchestra.run.smithi107.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-7531-g47c1c547-1jammy [180 kB] 2023-12-04T05:36:23.529 INFO:teuthology.orchestra.run.smithi107.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-7531-g47c1c547-1jammy [63.5 kB] 2023-12-04T05:36:23.529 INFO:teuthology.orchestra.run.smithi195.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-12-04T05:36:23.530 INFO:teuthology.orchestra.run.smithi107.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-7531-g47c1c547-1jammy [330 kB] 2023-12-04T05:36:23.538 INFO:teuthology.orchestra.run.smithi195.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-12-04T05:36:23.550 INFO:teuthology.orchestra.run.smithi107.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-7531-g47c1c547-1jammy [6,233 kB] 2023-12-04T05:36:23.703 INFO:teuthology.orchestra.run.smithi195.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-12-04T05:36:23.759 INFO:teuthology.orchestra.run.smithi195.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-7531-g47c1c547-1jammy [690 kB] 2023-12-04T05:36:23.830 INFO:teuthology.orchestra.run.smithi195.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-7531-g47c1c547-1jammy [356 kB] 2023-12-04T05:36:23.847 INFO:teuthology.orchestra.run.smithi107.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-7531-g47c1c547-1jammy [112 kB] 2023-12-04T05:36:23.849 INFO:teuthology.orchestra.run.smithi107.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-7531-g47c1c547-1jammy [439 kB] 2023-12-04T05:36:23.872 INFO:teuthology.orchestra.run.smithi195.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-7531-g47c1c547-1jammy [32.8 kB] 2023-12-04T05:36:23.872 INFO:teuthology.orchestra.run.smithi107.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-7531-g47c1c547-1jammy [22.6 MB] 2023-12-04T05:36:23.876 INFO:teuthology.orchestra.run.smithi195.stdout:Get:18 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-7531-g47c1c547-1jammy [180 kB] 2023-12-04T05:36:23.885 INFO:teuthology.orchestra.run.smithi195.stdout:Get:19 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-7531-g47c1c547-1jammy [63.5 kB] 2023-12-04T05:36:23.889 INFO:teuthology.orchestra.run.smithi195.stdout:Get:20 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-7531-g47c1c547-1jammy [330 kB] 2023-12-04T05:36:23.929 INFO:teuthology.orchestra.run.smithi195.stdout:Get:21 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-7531-g47c1c547-1jammy [6,233 kB] 2023-12-04T05:36:24.275 INFO:teuthology.orchestra.run.smithi195.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2023-12-04T05:36:24.290 INFO:teuthology.orchestra.run.smithi195.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2023-12-04T05:36:24.306 INFO:teuthology.orchestra.run.smithi195.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2023-12-04T05:36:24.314 INFO:teuthology.orchestra.run.smithi195.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2023-12-04T05:36:24.315 INFO:teuthology.orchestra.run.smithi195.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2023-12-04T05:36:24.331 INFO:teuthology.orchestra.run.smithi195.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] 2023-12-04T05:36:24.369 INFO:teuthology.orchestra.run.smithi195.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2023-12-04T05:36:24.382 INFO:teuthology.orchestra.run.smithi195.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-12-04T05:36:24.386 INFO:teuthology.orchestra.run.smithi195.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-12-04T05:36:24.387 INFO:teuthology.orchestra.run.smithi195.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-12-04T05:36:24.557 INFO:teuthology.orchestra.run.smithi195.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-12-04T05:36:24.620 INFO:teuthology.orchestra.run.smithi195.stdout:Get:33 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-7531-g47c1c547-1jammy [112 kB] 2023-12-04T05:36:24.622 INFO:teuthology.orchestra.run.smithi195.stdout:Get:34 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-7531-g47c1c547-1jammy [439 kB] 2023-12-04T05:36:24.640 INFO:teuthology.orchestra.run.smithi195.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-12-04T05:36:24.655 INFO:teuthology.orchestra.run.smithi195.stdout:Get:36 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-7531-g47c1c547-1jammy [22.6 MB] 2023-12-04T05:36:24.660 INFO:teuthology.orchestra.run.smithi195.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-12-04T05:36:24.721 INFO:teuthology.orchestra.run.smithi195.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-12-04T05:36:24.759 INFO:teuthology.orchestra.run.smithi195.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-12-04T05:36:24.780 INFO:teuthology.orchestra.run.smithi195.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-12-04T05:36:24.807 INFO:teuthology.orchestra.run.smithi195.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-12-04T05:36:24.994 INFO:teuthology.orchestra.run.smithi195.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-12-04T05:36:25.008 INFO:teuthology.orchestra.run.smithi195.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-12-04T05:36:25.015 INFO:teuthology.orchestra.run.smithi195.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-12-04T05:36:25.020 INFO:teuthology.orchestra.run.smithi195.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-12-04T05:36:25.082 INFO:teuthology.orchestra.run.smithi107.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-7531-g47c1c547-1jammy [6,900 kB] 2023-12-04T05:36:25.205 INFO:teuthology.orchestra.run.smithi195.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-12-04T05:36:25.288 INFO:teuthology.orchestra.run.smithi195.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-12-04T05:36:25.303 INFO:teuthology.orchestra.run.smithi195.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-12-04T05:36:25.350 INFO:teuthology.orchestra.run.smithi107.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-7531-g47c1c547-1jammy [238 kB] 2023-12-04T05:36:25.359 INFO:teuthology.orchestra.run.smithi107.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-7531-g47c1c547-1jammy [124 kB] 2023-12-04T05:36:25.365 INFO:teuthology.orchestra.run.smithi107.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-7531-g47c1c547-1jammy [1,496 kB] 2023-12-04T05:36:25.440 INFO:teuthology.orchestra.run.smithi107.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-7531-g47c1c547-1jammy [9,957 kB] 2023-12-04T05:36:25.488 INFO:teuthology.orchestra.run.smithi195.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-12-04T05:36:25.540 INFO:teuthology.orchestra.run.smithi195.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-12-04T05:36:25.554 INFO:teuthology.orchestra.run.smithi195.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-12-04T05:36:25.712 INFO:teuthology.orchestra.run.smithi195.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-12-04T05:36:25.713 INFO:teuthology.orchestra.run.smithi195.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-12-04T05:36:25.718 INFO:teuthology.orchestra.run.smithi195.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-12-04T05:36:25.722 INFO:teuthology.orchestra.run.smithi195.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-12-04T05:36:25.737 INFO:teuthology.orchestra.run.smithi195.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-12-04T05:36:25.922 INFO:teuthology.orchestra.run.smithi195.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-12-04T05:36:25.941 INFO:teuthology.orchestra.run.smithi107.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-7531-g47c1c547-1jammy [31.9 MB] 2023-12-04T05:36:26.186 INFO:teuthology.orchestra.run.smithi195.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2023-12-04T05:36:26.300 INFO:teuthology.orchestra.run.smithi195.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-12-04T05:36:26.346 INFO:teuthology.orchestra.run.smithi195.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-12-04T05:36:26.427 INFO:teuthology.orchestra.run.smithi195.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-12-04T05:36:26.439 INFO:teuthology.orchestra.run.smithi195.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-12-04T05:36:26.452 INFO:teuthology.orchestra.run.smithi195.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-12-04T05:36:26.493 INFO:teuthology.orchestra.run.smithi195.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-12-04T05:36:26.754 INFO:teuthology.orchestra.run.smithi195.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-12-04T05:36:26.756 INFO:teuthology.orchestra.run.smithi195.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-12-04T05:36:26.758 INFO:teuthology.orchestra.run.smithi195.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-12-04T05:36:26.842 INFO:teuthology.orchestra.run.smithi195.stdout:Get:68 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-7531-g47c1c547-1jammy [6,900 kB] 2023-12-04T05:36:26.922 INFO:teuthology.orchestra.run.smithi195.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-12-04T05:36:27.250 INFO:teuthology.orchestra.run.smithi195.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-12-04T05:36:27.532 INFO:teuthology.orchestra.run.smithi195.stdout:Get:71 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-7531-g47c1c547-1jammy [238 kB] 2023-12-04T05:36:27.562 INFO:teuthology.orchestra.run.smithi195.stdout:Get:72 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-7531-g47c1c547-1jammy [124 kB] 2023-12-04T05:36:27.569 INFO:teuthology.orchestra.run.smithi195.stdout:Get:73 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-7531-g47c1c547-1jammy [1,496 kB] 2023-12-04T05:36:27.650 INFO:teuthology.orchestra.run.smithi107.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-7531-g47c1c547-1jammy [14.0 kB] 2023-12-04T05:36:27.651 INFO:teuthology.orchestra.run.smithi107.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-7531-g47c1c547-1jammy [839 kB] 2023-12-04T05:36:27.654 INFO:teuthology.orchestra.run.smithi195.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-12-04T05:36:27.665 INFO:teuthology.orchestra.run.smithi195.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-12-04T05:36:27.666 INFO:teuthology.orchestra.run.smithi195.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-12-04T05:36:27.666 INFO:teuthology.orchestra.run.smithi107.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-7531-g47c1c547-1jammy [2,144 kB] 2023-12-04T05:36:27.674 INFO:teuthology.orchestra.run.smithi195.stdout:Get:77 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-7531-g47c1c547-1jammy [9,957 kB] 2023-12-04T05:36:27.704 INFO:teuthology.orchestra.run.smithi107.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-7531-g47c1c547-1jammy [802 kB] 2023-12-04T05:36:27.715 INFO:teuthology.orchestra.run.smithi107.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-7531-g47c1c547-1jammy [139 kB] 2023-12-04T05:36:27.717 INFO:teuthology.orchestra.run.smithi107.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-7531-g47c1c547-1jammy [1,900 kB] 2023-12-04T05:36:27.734 INFO:teuthology.orchestra.run.smithi195.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-12-04T05:36:27.735 INFO:teuthology.orchestra.run.smithi195.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-12-04T05:36:27.737 INFO:teuthology.orchestra.run.smithi195.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-12-04T05:36:27.738 INFO:teuthology.orchestra.run.smithi195.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-12-04T05:36:27.739 INFO:teuthology.orchestra.run.smithi195.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-12-04T05:36:27.772 INFO:teuthology.orchestra.run.smithi107.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-7531-g47c1c547-1jammy [8,625 kB] 2023-12-04T05:36:27.911 INFO:teuthology.orchestra.run.smithi195.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-12-04T05:36:28.006 INFO:teuthology.orchestra.run.smithi195.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-12-04T05:36:28.101 INFO:teuthology.orchestra.run.smithi195.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-12-04T05:36:28.192 INFO:teuthology.orchestra.run.smithi107.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-7531-g47c1c547-1jammy [14.2 kB] 2023-12-04T05:36:28.193 INFO:teuthology.orchestra.run.smithi107.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-7531-g47c1c547-1jammy [48.7 MB] 2023-12-04T05:36:28.345 INFO:teuthology.orchestra.run.smithi195.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-12-04T05:36:28.387 INFO:teuthology.orchestra.run.smithi195.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-12-04T05:36:28.412 INFO:teuthology.orchestra.run.smithi195.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-12-04T05:36:28.488 INFO:teuthology.orchestra.run.smithi195.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-12-04T05:36:28.581 INFO:teuthology.orchestra.run.smithi195.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-12-04T05:36:28.664 INFO:teuthology.orchestra.run.smithi195.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-12-04T05:36:28.664 INFO:teuthology.orchestra.run.smithi195.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-7531-g47c1c547-1jammy [31.9 MB] 2023-12-04T05:36:28.664 INFO:teuthology.orchestra.run.smithi195.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2023-12-04T05:36:28.752 INFO:teuthology.orchestra.run.smithi195.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2023-12-04T05:36:28.917 INFO:teuthology.orchestra.run.smithi195.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-12-04T05:36:29.448 INFO:teuthology.orchestra.run.smithi195.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2023-12-04T05:36:29.457 INFO:teuthology.orchestra.run.smithi195.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-12-04T05:36:29.493 INFO:teuthology.orchestra.run.smithi195.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-12-04T05:36:29.493 INFO:teuthology.orchestra.run.smithi195.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-12-04T05:36:29.500 INFO:teuthology.orchestra.run.smithi195.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-12-04T05:36:29.502 INFO:teuthology.orchestra.run.smithi195.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-12-04T05:36:29.544 INFO:teuthology.orchestra.run.smithi195.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-12-04T05:36:29.552 INFO:teuthology.orchestra.run.smithi195.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-12-04T05:36:29.624 INFO:teuthology.orchestra.run.smithi195.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-12-04T05:36:29.625 INFO:teuthology.orchestra.run.smithi195.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-12-04T05:36:29.791 INFO:teuthology.orchestra.run.smithi195.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-12-04T05:36:30.226 INFO:teuthology.orchestra.run.smithi195.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-12-04T05:36:30.241 INFO:teuthology.orchestra.run.smithi195.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-12-04T05:36:30.867 INFO:teuthology.orchestra.run.smithi107.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-7531-g47c1c547-1jammy [121 kB] 2023-12-04T05:36:30.869 INFO:teuthology.orchestra.run.smithi107.stdout:Get:119 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-7531-g47c1c547-1jammy [225 kB] 2023-12-04T05:36:30.874 INFO:teuthology.orchestra.run.smithi107.stdout:Get:120 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-7531-g47c1c547-1jammy [34.0 kB] 2023-12-04T05:36:30.875 INFO:teuthology.orchestra.run.smithi107.stdout:Get:121 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-7531-g47c1c547-1jammy [23.4 kB] 2023-12-04T05:36:30.875 INFO:teuthology.orchestra.run.smithi107.stdout:Get:122 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-7531-g47c1c547-1jammy [38.3 kB] 2023-12-04T05:36:30.876 INFO:teuthology.orchestra.run.smithi107.stdout:Get:123 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-7531-g47c1c547-1jammy [12.3 MB] 2023-12-04T05:36:31.315 INFO:teuthology.orchestra.run.smithi107.stdout:Get:124 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-7531-g47c1c547-1jammy [91.4 kB] 2023-12-04T05:36:31.545 INFO:teuthology.orchestra.run.smithi195.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-7531-g47c1c547-1jammy [14.0 kB] 2023-12-04T05:36:31.546 INFO:teuthology.orchestra.run.smithi195.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-7531-g47c1c547-1jammy [839 kB] 2023-12-04T05:36:31.562 INFO:teuthology.orchestra.run.smithi195.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-7531-g47c1c547-1jammy [2,144 kB] 2023-12-04T05:36:31.639 INFO:teuthology.orchestra.run.smithi195.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-7531-g47c1c547-1jammy [802 kB] 2023-12-04T05:36:31.681 INFO:teuthology.orchestra.run.smithi195.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-7531-g47c1c547-1jammy [139 kB] 2023-12-04T05:36:31.684 INFO:teuthology.orchestra.run.smithi195.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-7531-g47c1c547-1jammy [1,900 kB] 2023-12-04T05:36:31.761 INFO:teuthology.orchestra.run.smithi195.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-7531-g47c1c547-1jammy [8,625 kB] 2023-12-04T05:36:31.925 INFO:teuthology.orchestra.run.smithi107.stdout:Fetched 187 MB in 8s (22.4 MB/s) 2023-12-04T05:36:32.061 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pyperclip. 2023-12-04T05:36:32.145 INFO:teuthology.orchestra.run.smithi195.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-7531-g47c1c547-1jammy [14.2 kB] 2023-12-04T05:36:32.145 INFO:teuthology.orchestra.run.smithi195.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-7531-g47c1c547-1jammy [48.7 MB] 2023-12-04T05:36:32.155 INFO:teuthology.orchestra.run.smithi107.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165983 files and directories currently installed.) 2023-12-04T05:36:32.161 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-12-04T05:36:32.201 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-12-04T05:36:32.504 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-wcwidth. 2023-12-04T05:36:32.531 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-12-04T05:36:32.610 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-04T05:36:33.041 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-cmd2. 2023-12-04T05:36:33.067 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-12-04T05:36:33.132 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-12-04T05:36:33.671 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-12-04T05:36:33.697 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-12-04T05:36:33.779 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-04T05:36:34.133 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-12-04T05:36:34.159 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-12-04T05:36:34.200 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-04T05:36:34.462 INFO:teuthology.orchestra.run.smithi195.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-7531-g47c1c547-1jammy [121 kB] 2023-12-04T05:36:34.464 INFO:teuthology.orchestra.run.smithi195.stdout:Get:119 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-7531-g47c1c547-1jammy [225 kB] 2023-12-04T05:36:34.469 INFO:teuthology.orchestra.run.smithi195.stdout:Get:120 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-7531-g47c1c547-1jammy [34.0 kB] 2023-12-04T05:36:34.469 INFO:teuthology.orchestra.run.smithi195.stdout:Get:121 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-7531-g47c1c547-1jammy [23.4 kB] 2023-12-04T05:36:34.470 INFO:teuthology.orchestra.run.smithi195.stdout:Get:122 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-7531-g47c1c547-1jammy [38.3 kB] 2023-12-04T05:36:34.472 INFO:teuthology.orchestra.run.smithi195.stdout:Get:123 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-7531-g47c1c547-1jammy [12.3 MB] 2023-12-04T05:36:34.544 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-12-04T05:36:34.570 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-12-04T05:36:34.611 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-04T05:36:34.897 INFO:teuthology.orchestra.run.smithi195.stdout:Get:124 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-7531-g47c1c547-1jammy [91.4 kB] 2023-12-04T05:36:34.981 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-12-04T05:36:35.008 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-04T05:36:35.056 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T05:36:35.468 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-12-04T05:36:35.494 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-04T05:36:35.496 INFO:teuthology.orchestra.run.smithi195.stdout:Fetched 187 MB in 12s (15.2 MB/s) 2023-12-04T05:36:35.534 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T05:36:35.628 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-pyperclip. 2023-12-04T05:36:35.720 INFO:teuthology.orchestra.run.smithi195.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165983 files and directories currently installed.) 2023-12-04T05:36:35.726 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-12-04T05:36:35.769 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-12-04T05:36:35.863 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-12-04T05:36:35.889 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-04T05:36:35.921 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T05:36:36.071 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-wcwidth. 2023-12-04T05:36:36.097 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-12-04T05:36:36.162 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-04T05:36:36.249 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-12-04T05:36:36.276 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-12-04T05:36:36.316 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-04T05:36:36.465 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-cmd2. 2023-12-04T05:36:36.492 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-12-04T05:36:36.532 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-12-04T05:36:36.661 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package librados2. 2023-12-04T05:36:36.687 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../010-librados2_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:36.727 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking librados2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:36.885 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-12-04T05:36:36.911 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-12-04T05:36:36.977 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-04T05:36:37.172 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package librbd1. 2023-12-04T05:36:37.198 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../011-librbd1_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:37.238 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking librbd1 (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:37.314 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-12-04T05:36:37.340 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-12-04T05:36:37.380 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-04T05:36:37.675 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libcephfs2. 2023-12-04T05:36:37.705 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../012-libcephfs2_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:37.708 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-12-04T05:36:37.734 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-12-04T05:36:37.775 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-04T05:36:37.866 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libcephfs2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:38.144 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-rados. 2023-12-04T05:36:38.153 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-12-04T05:36:38.159 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../013-python3-rados_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:38.180 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-04T05:36:38.194 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-rados (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:38.228 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T05:36:38.505 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-12-04T05:36:38.532 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../014-python3-ceph-argparse_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-12-04T05:36:38.572 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-ceph-argparse (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:38.640 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-12-04T05:36:38.666 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-04T05:36:38.707 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T05:36:38.824 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-cephfs. 2023-12-04T05:36:38.840 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../015-python3-cephfs_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:38.883 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-cephfs (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:39.068 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-12-04T05:36:39.095 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-04T05:36:39.134 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T05:36:39.202 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-ceph-common. 2023-12-04T05:36:39.229 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../016-python3-ceph-common_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-12-04T05:36:39.269 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-ceph-common (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:39.589 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-prettytable. 2023-12-04T05:36:39.615 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2023-12-04T05:36:39.638 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-12-04T05:36:39.647 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-12-04T05:36:39.664 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-12-04T05:36:39.730 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-04T05:36:39.883 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-rbd. 2023-12-04T05:36:39.910 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../018-python3-rbd_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:39.950 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-rbd (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:40.065 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package librados2. 2023-12-04T05:36:40.092 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../010-librados2_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:40.132 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking librados2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:40.328 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-12-04T05:36:40.355 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-12-04T05:36:40.394 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-04T05:36:40.585 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package librbd1. 2023-12-04T05:36:40.612 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../011-librbd1_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:40.651 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking librbd1 (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:40.664 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libreadline-dev:amd64. 2023-12-04T05:36:40.679 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2023-12-04T05:36:40.714 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2023-12-04T05:36:41.000 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2023-12-04T05:36:41.027 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2023-12-04T05:36:41.054 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libcephfs2. 2023-12-04T05:36:41.068 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-04T05:36:41.081 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../012-libcephfs2_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:41.113 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libcephfs2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:41.379 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package lua5.1. 2023-12-04T05:36:41.399 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-rados. 2023-12-04T05:36:41.405 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2023-12-04T05:36:41.425 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../013-python3-rados_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:41.454 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2023-12-04T05:36:41.465 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-rados (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:41.766 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package lua-any. 2023-12-04T05:36:41.792 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2023-12-04T05:36:41.810 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-12-04T05:36:41.832 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking lua-any (27ubuntu1) ... 2023-12-04T05:36:41.836 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../014-python3-ceph-argparse_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-12-04T05:36:41.876 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-ceph-argparse (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:42.051 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package zip. 2023-12-04T05:36:42.078 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2023-12-04T05:36:42.118 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking zip (3.0-12build2) ... 2023-12-04T05:36:42.154 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-cephfs. 2023-12-04T05:36:42.180 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../015-python3-cephfs_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:42.220 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-cephfs (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:42.505 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package unzip. 2023-12-04T05:36:42.531 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.1_amd64.deb ... 2023-12-04T05:36:42.531 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-ceph-common. 2023-12-04T05:36:42.557 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../016-python3-ceph-common_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-12-04T05:36:42.580 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking unzip (6.0-26ubuntu3.1) ... 2023-12-04T05:36:42.598 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-ceph-common (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:42.943 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-prettytable. 2023-12-04T05:36:42.969 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2023-12-04T05:36:43.009 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-12-04T05:36:43.017 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package luarocks. 2023-12-04T05:36:43.043 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2023-12-04T05:36:43.084 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2023-12-04T05:36:43.290 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-rbd. 2023-12-04T05:36:43.317 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../018-python3-rbd_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:43.362 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-rbd (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:43.451 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package librgw2. 2023-12-04T05:36:43.477 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../027-librgw2_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:43.517 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking librgw2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:43.656 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-12-04T05:36:43.683 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-12-04T05:36:43.714 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-04T05:36:43.967 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libreadline-dev:amd64. 2023-12-04T05:36:43.994 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2023-12-04T05:36:44.034 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2023-12-04T05:36:44.094 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-rgw. 2023-12-04T05:36:44.121 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../028-python3-rgw_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:44.169 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-rgw (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:44.328 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2023-12-04T05:36:44.344 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2023-12-04T05:36:44.386 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-04T05:36:44.573 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package liboath0:amd64. 2023-12-04T05:36:44.599 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2023-12-04T05:36:44.641 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-12-04T05:36:44.740 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package lua5.1. 2023-12-04T05:36:44.766 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2023-12-04T05:36:44.806 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2023-12-04T05:36:44.942 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libradosstriper1. 2023-12-04T05:36:44.958 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../030-libradosstriper1_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:45.000 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libradosstriper1 (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:45.093 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package lua-any. 2023-12-04T05:36:45.119 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2023-12-04T05:36:45.159 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking lua-any (27ubuntu1) ... 2023-12-04T05:36:45.412 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package zip. 2023-12-04T05:36:45.439 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2023-12-04T05:36:45.479 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking zip (3.0-12build2) ... 2023-12-04T05:36:45.537 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-common. 2023-12-04T05:36:45.563 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../031-ceph-common_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:45.612 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-common (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:45.773 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package unzip. 2023-12-04T05:36:45.800 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.1_amd64.deb ... 2023-12-04T05:36:45.840 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking unzip (6.0-26ubuntu3.1) ... 2023-12-04T05:36:46.194 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package luarocks. 2023-12-04T05:36:46.222 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2023-12-04T05:36:46.252 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2023-12-04T05:36:46.626 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package librgw2. 2023-12-04T05:36:46.642 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../027-librgw2_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:46.676 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking librgw2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:46.959 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-base. 2023-12-04T05:36:46.986 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../032-ceph-base_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:47.043 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-base (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:47.212 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-rgw. 2023-12-04T05:36:47.236 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../028-python3-rgw_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:47.279 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-rgw (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:47.623 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package liboath0:amd64. 2023-12-04T05:36:47.650 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2023-12-04T05:36:47.690 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-12-04T05:36:47.772 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-12-04T05:36:47.799 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-12-04T05:36:47.839 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-12-04T05:36:48.001 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libradosstriper1. 2023-12-04T05:36:48.028 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../030-libradosstriper1_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:48.068 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libradosstriper1 (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:48.442 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-cheroot. 2023-12-04T05:36:48.446 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-common. 2023-12-04T05:36:48.468 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-12-04T05:36:48.472 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../031-ceph-common_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:48.513 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-common (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:48.516 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-04T05:36:48.895 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-12-04T05:36:48.921 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-12-04T05:36:48.970 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-12-04T05:36:49.298 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-jaraco.text. 2023-12-04T05:36:49.324 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2023-12-04T05:36:49.364 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-12-04T05:36:49.668 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-12-04T05:36:49.694 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-12-04T05:36:49.734 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-12-04T05:36:49.910 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-base. 2023-12-04T05:36:49.937 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../032-ceph-base_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:49.986 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-base (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:50.070 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-tempora. 2023-12-04T05:36:50.097 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2023-12-04T05:36:50.137 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-12-04T05:36:50.390 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-portend. 2023-12-04T05:36:50.417 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2023-12-04T05:36:50.457 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-12-04T05:36:50.731 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-12-04T05:36:50.757 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-12-04T05:36:50.768 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-12-04T05:36:50.795 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2023-12-04T05:36:50.806 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-12-04T05:36:50.835 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-12-04T05:36:51.100 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-cheroot. 2023-12-04T05:36:51.126 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-12-04T05:36:51.146 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-cherrypy3. 2023-12-04T05:36:51.166 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-04T05:36:51.173 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2023-12-04T05:36:51.213 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-12-04T05:36:51.428 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-12-04T05:36:51.451 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-12-04T05:36:51.486 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-12-04T05:36:51.525 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-natsort. 2023-12-04T05:36:51.541 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2023-12-04T05:36:51.574 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-12-04T05:36:51.791 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-jaraco.text. 2023-12-04T05:36:51.815 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2023-12-04T05:36:51.844 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-logutils. 2023-12-04T05:36:51.855 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-12-04T05:36:51.871 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2023-12-04T05:36:51.911 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-12-04T05:36:52.175 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-12-04T05:36:52.201 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-12-04T05:36:52.241 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-12-04T05:36:52.247 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-markupsafe. 2023-12-04T05:36:52.274 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-12-04T05:36:52.314 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-12-04T05:36:52.528 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-tempora. 2023-12-04T05:36:52.554 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2023-12-04T05:36:52.586 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-12-04T05:36:52.617 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-mako. 2023-12-04T05:36:52.644 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-12-04T05:36:52.684 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-04T05:36:52.859 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-portend. 2023-12-04T05:36:52.882 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2023-12-04T05:36:52.922 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-12-04T05:36:52.979 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-simplegeneric. 2023-12-04T05:36:52.994 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2023-12-04T05:36:53.029 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-12-04T05:36:53.225 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-12-04T05:36:53.251 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2023-12-04T05:36:53.291 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-12-04T05:36:53.299 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-singledispatch. 2023-12-04T05:36:53.325 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-12-04T05:36:53.365 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-12-04T05:36:53.595 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-cherrypy3. 2023-12-04T05:36:53.621 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2023-12-04T05:36:53.661 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-12-04T05:36:53.668 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-webob. 2023-12-04T05:36:53.694 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-12-04T05:36:53.734 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-12-04T05:36:54.015 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-natsort. 2023-12-04T05:36:54.041 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2023-12-04T05:36:54.081 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-12-04T05:36:54.113 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-waitress. 2023-12-04T05:36:54.139 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-12-04T05:36:54.183 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-04T05:36:54.418 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-logutils. 2023-12-04T05:36:54.444 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2023-12-04T05:36:54.457 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-tempita. 2023-12-04T05:36:54.484 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-12-04T05:36:54.493 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-12-04T05:36:54.516 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-12-04T05:36:54.770 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-markupsafe. 2023-12-04T05:36:54.797 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-12-04T05:36:54.827 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-paste. 2023-12-04T05:36:54.837 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-12-04T05:36:54.853 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-12-04T05:36:54.893 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-12-04T05:36:55.106 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-mako. 2023-12-04T05:36:55.133 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-12-04T05:36:55.173 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-04T05:36:55.205 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-12-04T05:36:55.221 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-12-04T05:36:55.255 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-12-04T05:36:55.502 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-simplegeneric. 2023-12-04T05:36:55.528 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2023-12-04T05:36:55.550 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pastedeploy. 2023-12-04T05:36:55.568 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-12-04T05:36:55.576 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2023-12-04T05:36:55.616 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-12-04T05:36:55.871 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-singledispatch. 2023-12-04T05:36:55.897 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-12-04T05:36:55.937 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-12-04T05:36:56.206 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-webob. 2023-12-04T05:36:56.222 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-12-04T05:36:56.256 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-12-04T05:36:56.428 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-webtest. 2023-12-04T05:36:56.455 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2023-12-04T05:36:56.585 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-waitress. 2023-12-04T05:36:56.611 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-12-04T05:36:56.663 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-04T05:36:56.745 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-12-04T05:36:56.929 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-tempita. 2023-12-04T05:36:56.958 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-12-04T05:36:56.987 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-12-04T05:36:57.057 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pecan. 2023-12-04T05:36:57.083 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-12-04T05:36:57.123 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-12-04T05:36:57.274 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-paste. 2023-12-04T05:36:57.300 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-12-04T05:36:57.340 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-12-04T05:36:57.520 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-certifi. 2023-12-04T05:36:57.546 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2023-12-04T05:36:57.594 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-12-04T05:36:57.627 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-12-04T05:36:57.642 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-12-04T05:36:57.676 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-12-04T05:36:57.948 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-idna. 2023-12-04T05:36:57.974 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2023-12-04T05:36:57.988 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-pastedeploy. 2023-12-04T05:36:58.014 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2023-12-04T05:36:58.015 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-idna (3.3-1) ... 2023-12-04T05:36:58.054 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-12-04T05:36:58.329 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-urllib3. 2023-12-04T05:36:58.352 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2023-12-04T05:36:58.357 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-webtest. 2023-12-04T05:36:58.386 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2023-12-04T05:36:58.392 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-04T05:36:58.424 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-12-04T05:36:58.738 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-requests. 2023-12-04T05:36:58.760 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-pecan. 2023-12-04T05:36:58.764 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-12-04T05:36:58.787 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-12-04T05:36:58.806 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-04T05:36:58.826 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-12-04T05:36:59.116 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-werkzeug. 2023-12-04T05:36:59.142 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-12-04T05:36:59.182 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-04T05:36:59.190 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-certifi. 2023-12-04T05:36:59.216 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2023-12-04T05:36:59.256 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-12-04T05:36:59.485 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-12-04T05:36:59.501 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-12-04T05:36:59.527 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-idna. 2023-12-04T05:36:59.541 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2023-12-04T05:36:59.544 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mgr-modules-core (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:36:59.575 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-idna (3.3-1) ... 2023-12-04T05:36:59.889 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-urllib3. 2023-12-04T05:36:59.918 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2023-12-04T05:36:59.932 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-12-04T05:36:59.955 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-04T05:36:59.959 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:36:59.998 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:00.284 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-requests. 2023-12-04T05:37:00.310 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-12-04T05:37:00.318 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-bcrypt. 2023-12-04T05:37:00.345 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-12-04T05:37:00.350 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-04T05:37:00.385 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-12-04T05:37:00.661 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-werkzeug. 2023-12-04T05:37:00.671 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-openssl. 2023-12-04T05:37:00.688 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-12-04T05:37:00.698 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2023-12-04T05:37:00.728 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-04T05:37:00.738 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-12-04T05:37:01.031 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-12-04T05:37:01.050 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mgr. 2023-12-04T05:37:01.058 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-12-04T05:37:01.076 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../065-ceph-mgr_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:01.097 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-mgr-modules-core (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:01.116 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mgr (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:01.435 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-12-04T05:37:01.451 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:01.502 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:01.519 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mon. 2023-12-04T05:37:01.546 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../066-ceph-mon_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:01.585 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mon (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:01.822 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-bcrypt. 2023-12-04T05:37:01.849 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-12-04T05:37:01.888 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-12-04T05:37:02.199 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-openssl. 2023-12-04T05:37:02.226 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2023-12-04T05:37:02.266 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-12-04T05:37:02.514 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-osd. 2023-12-04T05:37:02.541 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../067-ceph-osd_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:02.571 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-mgr. 2023-12-04T05:37:02.572 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-osd (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:02.595 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../065-ceph-mgr_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:02.635 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-mgr (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:03.030 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-mon. 2023-12-04T05:37:03.057 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../066-ceph-mon_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:03.098 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-mon (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:04.025 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-osd. 2023-12-04T05:37:04.052 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../067-ceph-osd_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:04.092 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-osd (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:04.592 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph. 2023-12-04T05:37:04.616 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../068-ceph_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:04.651 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:04.911 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-fuse. 2023-12-04T05:37:04.941 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../069-ceph-fuse_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:04.969 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-fuse (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:05.375 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mds. 2023-12-04T05:37:05.404 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../070-ceph-mds_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:05.439 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mds (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:05.834 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package cephadm. 2023-12-04T05:37:05.861 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../071-cephadm_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:05.901 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking cephadm (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:06.011 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph. 2023-12-04T05:37:06.038 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../068-ceph_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:06.078 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:06.162 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-asyncssh. 2023-12-04T05:37:06.190 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2023-12-04T05:37:06.220 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-12-04T05:37:06.397 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-fuse. 2023-12-04T05:37:06.424 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../069-ceph-fuse_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:06.464 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-fuse (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:06.533 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python-babel-localedata. 2023-12-04T05:37:06.547 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-12-04T05:37:06.581 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-04T05:37:06.800 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-mds. 2023-12-04T05:37:06.823 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../070-ceph-mds_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:06.858 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-mds (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:07.228 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package cephadm. 2023-12-04T05:37:07.254 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../071-cephadm_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:07.295 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking cephadm (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:07.595 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-babel. 2023-12-04T05:37:07.606 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-asyncssh. 2023-12-04T05:37:07.623 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-12-04T05:37:07.632 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2023-12-04T05:37:07.662 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-12-04T05:37:07.674 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-12-04T05:37:07.958 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python-babel-localedata. 2023-12-04T05:37:07.985 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-12-04T05:37:07.991 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-jinja2. 2023-12-04T05:37:08.018 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1_all.deb ... 2023-12-04T05:37:08.018 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-04T05:37:08.056 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-12-04T05:37:08.401 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-12-04T05:37:08.428 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-12-04T05:37:08.467 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mgr-cephadm (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:08.813 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-repoze.lru. 2023-12-04T05:37:08.840 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2023-12-04T05:37:08.879 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-12-04T05:37:09.046 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-babel. 2023-12-04T05:37:09.073 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-12-04T05:37:09.113 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-12-04T05:37:09.174 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-routes. 2023-12-04T05:37:09.195 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-12-04T05:37:09.224 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-12-04T05:37:09.424 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-jinja2. 2023-12-04T05:37:09.451 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1_all.deb ... 2023-12-04T05:37:09.491 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-12-04T05:37:09.527 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-12-04T05:37:09.543 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-12-04T05:37:09.577 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mgr-dashboard (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:09.818 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-12-04T05:37:09.845 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-12-04T05:37:09.885 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-mgr-cephadm (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:10.014 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-12-04T05:37:10.041 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-12-04T05:37:10.081 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-04T05:37:10.205 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-repoze.lru. 2023-12-04T05:37:10.232 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2023-12-04T05:37:10.271 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-12-04T05:37:10.567 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-joblib. 2023-12-04T05:37:10.574 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-routes. 2023-12-04T05:37:10.594 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-12-04T05:37:10.601 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-12-04T05:37:10.625 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-12-04T05:37:10.641 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-12-04T05:37:10.954 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-12-04T05:37:10.979 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-12-04T05:37:10.990 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-12-04T05:37:11.017 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-12-04T05:37:11.018 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-mgr-dashboard (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:11.150 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-12-04T05:37:11.414 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-12-04T05:37:11.429 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-12-04T05:37:11.444 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-sklearn. 2023-12-04T05:37:11.471 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-12-04T05:37:11.471 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-04T05:37:11.510 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-04T05:37:11.900 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-joblib. 2023-12-04T05:37:11.927 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-12-04T05:37:11.958 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-12-04T05:37:12.269 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-12-04T05:37:12.297 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-12-04T05:37:12.320 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-12-04T05:37:12.335 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:12.347 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-12-04T05:37:12.498 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-12-04T05:37:12.767 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-sklearn. 2023-12-04T05:37:12.793 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-12-04T05:37:12.825 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-04T05:37:13.373 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-cachetools. 2023-12-04T05:37:13.399 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2023-12-04T05:37:13.440 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-12-04T05:37:13.450 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-12-04T05:37:13.466 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-12-04T05:37:13.500 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:13.741 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pyasn1. 2023-12-04T05:37:13.769 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2023-12-04T05:37:13.816 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-12-04T05:37:14.127 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-12-04T05:37:14.155 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-12-04T05:37:14.185 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-12-04T05:37:14.420 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-cachetools. 2023-12-04T05:37:14.447 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2023-12-04T05:37:14.455 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-rsa. 2023-12-04T05:37:14.482 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2023-12-04T05:37:14.486 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-12-04T05:37:14.513 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-rsa (4.8-1) ... 2023-12-04T05:37:14.790 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-pyasn1. 2023-12-04T05:37:14.817 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2023-12-04T05:37:14.856 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-12-04T05:37:15.042 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-google-auth. 2023-12-04T05:37:15.069 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2023-12-04T05:37:15.186 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-12-04T05:37:15.212 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-12-04T05:37:15.225 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-12-04T05:37:15.250 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-12-04T05:37:15.545 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-12-04T05:37:15.572 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-12-04T05:37:15.611 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-04T05:37:15.713 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-rsa. 2023-12-04T05:37:15.741 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2023-12-04T05:37:15.787 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-rsa (4.8-1) ... 2023-12-04T05:37:15.905 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-websocket. 2023-12-04T05:37:15.933 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2023-12-04T05:37:15.980 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-12-04T05:37:16.091 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-google-auth. 2023-12-04T05:37:16.118 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2023-12-04T05:37:16.149 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-12-04T05:37:16.284 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-kubernetes. 2023-12-04T05:37:16.311 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-12-04T05:37:16.342 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-04T05:37:16.419 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-12-04T05:37:16.446 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-12-04T05:37:16.485 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-04T05:37:16.796 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-websocket. 2023-12-04T05:37:16.823 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2023-12-04T05:37:16.863 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-12-04T05:37:17.013 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-12-04T05:37:17.041 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-12-04T05:37:17.080 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:17.175 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-kubernetes. 2023-12-04T05:37:17.202 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-12-04T05:37:17.241 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-04T05:37:17.416 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libonig5:amd64. 2023-12-04T05:37:17.443 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-12-04T05:37:17.482 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-04T05:37:17.752 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libjq1:amd64. 2023-12-04T05:37:17.768 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-12-04T05:37:17.802 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-04T05:37:17.930 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-12-04T05:37:17.958 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-12-04T05:37:17.996 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:18.071 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package jq. 2023-12-04T05:37:18.087 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-12-04T05:37:18.129 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-12-04T05:37:18.282 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libonig5:amd64. 2023-12-04T05:37:18.310 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-12-04T05:37:18.349 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-04T05:37:18.390 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package socat. 2023-12-04T05:37:18.418 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-12-04T05:37:18.457 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-12-04T05:37:18.669 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libjq1:amd64. 2023-12-04T05:37:18.696 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-12-04T05:37:18.735 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-04T05:37:18.751 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package xmlstarlet. 2023-12-04T05:37:18.767 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-12-04T05:37:18.810 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-12-04T05:37:18.996 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package jq. 2023-12-04T05:37:19.021 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-12-04T05:37:19.063 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-12-04T05:37:19.188 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-test. 2023-12-04T05:37:19.204 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../099-ceph-test_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:19.247 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-test (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:19.315 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package socat. 2023-12-04T05:37:19.343 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-12-04T05:37:19.382 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-12-04T05:37:19.668 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package xmlstarlet. 2023-12-04T05:37:19.684 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-12-04T05:37:19.718 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-12-04T05:37:20.047 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-test. 2023-12-04T05:37:20.063 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../099-ceph-test_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:20.097 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-test (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:21.492 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-volume. 2023-12-04T05:37:21.520 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../100-ceph-volume_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-12-04T05:37:21.559 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-volume (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:22.004 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package cephfs-mirror. 2023-12-04T05:37:22.031 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../101-cephfs-mirror_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:22.072 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking cephfs-mirror (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:22.193 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-volume. 2023-12-04T05:37:22.220 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../100-ceph-volume_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-12-04T05:37:22.251 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-volume (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:22.345 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-colorama. 2023-12-04T05:37:22.356 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2023-12-04T05:37:22.390 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-12-04T05:37:22.614 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package cephfs-mirror. 2023-12-04T05:37:22.641 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../101-cephfs-mirror_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:22.667 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package cephfs-shell. 2023-12-04T05:37:22.680 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking cephfs-mirror (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:22.683 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../103-cephfs-shell_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-12-04T05:37:22.726 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking cephfs-shell (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:22.966 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-colorama. 2023-12-04T05:37:22.970 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package cephfs-top. 2023-12-04T05:37:22.998 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../104-cephfs-top_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-12-04T05:37:22.999 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2023-12-04T05:37:23.024 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-12-04T05:37:23.036 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking cephfs-top (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:23.251 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package cephfs-shell. 2023-12-04T05:37:23.276 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../103-cephfs-shell_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-12-04T05:37:23.288 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libcephfs-dev. 2023-12-04T05:37:23.304 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../105-libcephfs-dev_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:23.318 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking cephfs-shell (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:23.347 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libcephfs-dev (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:23.571 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package cephfs-top. 2023-12-04T05:37:23.599 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../104-cephfs-top_18.0.0-7531-g47c1c547-1jammy_all.deb ... 2023-12-04T05:37:23.638 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking cephfs-top (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:23.724 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package lua-socket:amd64. 2023-12-04T05:37:23.752 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2023-12-04T05:37:23.791 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-04T05:37:23.890 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libcephfs-dev. 2023-12-04T05:37:23.906 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../105-libcephfs-dev_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:23.948 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libcephfs-dev (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:24.070 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package lua-sec:amd64. 2023-12-04T05:37:24.086 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2023-12-04T05:37:24.119 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2023-12-04T05:37:24.268 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package lua-socket:amd64. 2023-12-04T05:37:24.295 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2023-12-04T05:37:24.334 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-04T05:37:24.507 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package nvme-cli. 2023-12-04T05:37:24.535 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-12-04T05:37:24.573 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-12-04T05:37:24.642 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package lua-sec:amd64. 2023-12-04T05:37:24.666 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2023-12-04T05:37:24.696 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2023-12-04T05:37:24.998 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package pkg-config. 2023-12-04T05:37:25.023 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2023-12-04T05:37:25.061 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2023-12-04T05:37:25.076 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package nvme-cli. 2023-12-04T05:37:25.103 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-12-04T05:37:25.142 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-12-04T05:37:25.322 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-12-04T05:37:25.339 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-12-04T05:37:25.381 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-12-04T05:37:25.505 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package pkg-config. 2023-12-04T05:37:25.533 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2023-12-04T05:37:25.572 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2023-12-04T05:37:25.850 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-12-04T05:37:25.875 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-iniconfig. 2023-12-04T05:37:25.877 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-12-04T05:37:25.903 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2023-12-04T05:37:25.916 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-12-04T05:37:25.950 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-12-04T05:37:26.286 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pastescript. 2023-12-04T05:37:26.311 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-iniconfig. 2023-12-04T05:37:26.315 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2023-12-04T05:37:26.339 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2023-12-04T05:37:26.352 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-12-04T05:37:26.377 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-12-04T05:37:26.681 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pluggy. 2023-12-04T05:37:26.688 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-pastescript. 2023-12-04T05:37:26.709 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2023-12-04T05:37:26.716 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2023-12-04T05:37:26.747 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-12-04T05:37:26.755 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-12-04T05:37:27.092 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-pluggy. 2023-12-04T05:37:27.093 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-psutil. 2023-12-04T05:37:27.119 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2023-12-04T05:37:27.120 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-12-04T05:37:27.158 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-12-04T05:37:27.158 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-12-04T05:37:27.427 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-psutil. 2023-12-04T05:37:27.455 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-12-04T05:37:27.494 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-12-04T05:37:27.512 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-py. 2023-12-04T05:37:27.540 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2023-12-04T05:37:27.586 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-py (1.10.0-1) ... 2023-12-04T05:37:27.814 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-py. 2023-12-04T05:37:27.842 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2023-12-04T05:37:27.880 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-py (1.10.0-1) ... 2023-12-04T05:37:27.915 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pygments. 2023-12-04T05:37:27.943 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-12-04T05:37:27.981 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-12-04T05:37:28.209 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-pygments. 2023-12-04T05:37:28.236 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-12-04T05:37:28.275 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-12-04T05:37:28.469 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pyinotify. 2023-12-04T05:37:28.496 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-12-04T05:37:28.534 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-12-04T05:37:28.737 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-pyinotify. 2023-12-04T05:37:28.765 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-12-04T05:37:28.787 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-toml. 2023-12-04T05:37:28.803 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2023-12-04T05:37:28.804 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-12-04T05:37:28.837 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-12-04T05:37:29.081 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-toml. 2023-12-04T05:37:29.098 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2023-12-04T05:37:29.131 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-12-04T05:37:29.148 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pytest. 2023-12-04T05:37:29.176 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-12-04T05:37:29.214 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-12-04T05:37:29.434 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-pytest. 2023-12-04T05:37:29.462 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-12-04T05:37:29.500 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-12-04T05:37:29.543 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-simplejson. 2023-12-04T05:37:29.571 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-12-04T05:37:29.609 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-12-04T05:37:29.804 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-simplejson. 2023-12-04T05:37:29.820 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-12-04T05:37:29.854 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-12-04T05:37:29.870 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-12-04T05:37:29.886 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-12-04T05:37:29.929 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-12-04T05:37:30.083 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-12-04T05:37:30.097 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-12-04T05:37:30.133 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-12-04T05:37:30.860 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package radosgw. 2023-12-04T05:37:30.889 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../122-radosgw_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:30.927 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking radosgw (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:30.966 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package radosgw. 2023-12-04T05:37:30.994 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../122-radosgw_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:31.032 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking radosgw (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:31.754 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package rbd-fuse. 2023-12-04T05:37:31.771 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../123-rbd-fuse_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:31.806 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking rbd-fuse (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:31.826 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package rbd-fuse. 2023-12-04T05:37:31.843 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../123-rbd-fuse_18.0.0-7531-g47c1c547-1jammy_amd64.deb ... 2023-12-04T05:37:31.884 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking rbd-fuse (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:32.080 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-12-04T05:37:32.197 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-12-04T05:37:32.334 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-04T05:37:32.438 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-12-04T05:37:32.441 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-04T05:37:32.565 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-12-04T05:37:32.672 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-12-04T05:37:32.810 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-12-04T05:37:33.034 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-12-04T05:37:33.163 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-12-04T05:37:33.530 INFO:teuthology.orchestra.run.smithi195.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-12-04T05:37:33.622 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up cephadm (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:33.667 INFO:teuthology.orchestra.run.smithi107.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-12-04T05:37:33.894 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up cephadm (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:34.071 INFO:teuthology.orchestra.run.smithi195.stdout:Adding system user cephadm....done 2023-12-04T05:37:34.243 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-04T05:37:34.427 INFO:teuthology.orchestra.run.smithi107.stdout:Adding system user cephadm....done 2023-12-04T05:37:34.528 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-12-04T05:37:34.582 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-04T05:37:34.754 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-12-04T05:37:34.858 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-12-04T05:37:34.880 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-12-04T05:37:35.100 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-12-04T05:37:35.131 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-12-04T05:37:35.226 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-12-04T05:37:35.391 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-12-04T05:37:35.478 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-12-04T05:37:35.533 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-py (1.10.0-1) ... 2023-12-04T05:37:35.851 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-12-04T05:37:35.945 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-12-04T05:37:36.055 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-py (1.10.0-1) ... 2023-12-04T05:37:36.220 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-12-04T05:37:36.348 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-12-04T05:37:36.488 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-12-04T05:37:36.700 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-12-04T05:37:36.739 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-12-04T05:37:36.959 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-12-04T05:37:37.015 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up unzip (6.0-26ubuntu3.1) ... 2023-12-04T05:37:37.210 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-12-04T05:37:37.292 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-12-04T05:37:37.478 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up unzip (6.0-26ubuntu3.1) ... 2023-12-04T05:37:37.526 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-12-04T05:37:37.722 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-12-04T05:37:37.777 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-12-04T05:37:37.973 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-12-04T05:37:38.037 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-ceph-argparse (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:38.232 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-12-04T05:37:38.305 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-04T05:37:38.431 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-04T05:37:38.483 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-ceph-argparse (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:38.557 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-12-04T05:37:38.734 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-04T05:37:38.816 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2023-12-04T05:37:38.860 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-04T05:37:38.917 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-12-04T05:37:38.986 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-12-04T05:37:39.151 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2023-12-04T05:37:39.270 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2023-12-04T05:37:39.387 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-12-04T05:37:39.505 INFO:teuthology.orchestra.run.smithi195.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2023-12-04T05:37:39.542 INFO:teuthology.orchestra.run.smithi195.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2023-12-04T05:37:39.605 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-04T05:37:39.622 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2023-12-04T05:37:39.731 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-12-04T05:37:40.030 INFO:teuthology.orchestra.run.smithi107.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2023-12-04T05:37:40.061 INFO:teuthology.orchestra.run.smithi107.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2023-12-04T05:37:40.090 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-04T05:37:40.116 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-04T05:37:40.216 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-12-04T05:37:40.217 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-12-04T05:37:40.484 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-12-04T05:37:40.578 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-04T05:37:40.728 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-12-04T05:37:40.743 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-12-04T05:37:40.995 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-12-04T05:37:41.095 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up zip (3.0-12build2) ... 2023-12-04T05:37:41.196 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-12-04T05:37:41.255 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-12-04T05:37:41.556 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up zip (3.0-12build2) ... 2023-12-04T05:37:41.682 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-12-04T05:37:41.824 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-12-04T05:37:42.084 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-12-04T05:37:42.210 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-12-04T05:37:42.334 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-12-04T05:37:42.336 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-04T05:37:42.585 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-12-04T05:37:42.612 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-12-04T05:37:42.711 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-12-04T05:37:42.836 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-04T05:37:42.971 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-12-04T05:37:43.121 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-12-04T05:37:43.197 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-12-04T05:37:43.506 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-12-04T05:37:43.574 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-04T05:37:43.740 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-12-04T05:37:43.858 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-12-04T05:37:44.091 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-04T05:37:44.168 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-04T05:37:44.375 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-12-04T05:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-12-04T05:37:44.677 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-04T05:37:44.745 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-12-04T05:37:44.846 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-ceph-common (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:45.044 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-12-04T05:37:45.139 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-idna (3.3-1) ... 2023-12-04T05:37:45.321 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-12-04T05:37:45.422 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-ceph-common (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:45.465 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-04T05:37:45.709 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-idna (3.3-1) ... 2023-12-04T05:37:46.007 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-04T05:37:46.646 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2023-12-04T05:37:46.914 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T05:37:47.049 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-toml (0.10.2-1) ... 2023-12-04T05:37:47.179 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2023-12-04T05:37:47.318 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-04T05:37:47.397 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T05:37:47.443 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-12-04T05:37:47.532 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-toml (0.10.2-1) ... 2023-12-04T05:37:47.560 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-04T05:37:47.791 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-04T05:37:47.836 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-12-04T05:37:47.900 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-12-04T05:37:48.026 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-04T05:37:48.071 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-12-04T05:37:48.305 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T05:37:48.336 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-12-04T05:37:48.431 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-12-04T05:37:48.621 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-12-04T05:37:48.749 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-12-04T05:37:48.871 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T05:37:48.989 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-12-04T05:37:49.009 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-12-04T05:37:49.268 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-12-04T05:37:49.282 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-12-04T05:37:49.516 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-12-04T05:37:49.519 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-12-04T05:37:49.753 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-12-04T05:37:49.759 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-12-04T05:37:50.004 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-04T05:37:50.019 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-12-04T05:37:50.122 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-12-04T05:37:50.269 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-12-04T05:37:50.398 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-04T05:37:50.524 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-04T05:37:50.528 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-04T05:37:50.630 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-12-04T05:37:50.817 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-12-04T05:37:50.905 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-04T05:37:51.031 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-04T05:37:51.126 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-12-04T05:37:51.324 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-12-04T05:37:51.362 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-04T05:37:51.470 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2023-12-04T05:37:51.571 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-12-04T05:37:51.609 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-12-04T05:37:51.660 INFO:teuthology.orchestra.run.smithi195.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-12-04T05:37:51.868 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-04T05:37:51.923 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-04T05:37:52.019 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2023-12-04T05:37:52.049 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-12-04T05:37:52.138 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-12-04T05:37:52.209 INFO:teuthology.orchestra.run.smithi107.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-12-04T05:37:52.447 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-04T05:37:52.450 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-12-04T05:37:52.556 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-12-04T05:37:52.706 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up lua-any (27ubuntu1) ... 2023-12-04T05:37:52.827 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-portend (3.0.0-1) ... 2023-12-04T05:37:52.924 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-12-04T05:37:53.078 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T05:37:53.166 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up lua-any (27ubuntu1) ... 2023-12-04T05:37:53.179 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-12-04T05:37:53.292 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-portend (3.0.0-1) ... 2023-12-04T05:37:53.489 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-12-04T05:37:53.554 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T05:37:53.678 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-12-04T05:37:53.773 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-04T05:37:53.995 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-12-04T05:37:54.033 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-12-04T05:37:54.236 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-rsa (4.8-1) ... 2023-12-04T05:37:54.322 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-04T05:37:54.495 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-12-04T05:37:54.597 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-12-04T05:37:54.747 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-04T05:37:54.800 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-rsa (4.8-1) ... 2023-12-04T05:37:55.006 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-12-04T05:37:55.068 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-12-04T05:37:55.327 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-04T05:37:55.383 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-12-04T05:37:55.587 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-12-04T05:37:55.676 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-12-04T05:37:55.973 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-12-04T05:37:56.070 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-04T05:37:56.171 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up librados2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:56.257 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-12-04T05:37:56.289 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libsqlite3-mod-ceph (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:56.415 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2023-12-04T05:37:56.601 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-04T05:37:56.727 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up librados2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:56.768 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libcephfs2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:56.869 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libsqlite3-mod-ceph (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:56.894 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libradosstriper1 (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:56.996 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2023-12-04T05:37:57.012 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-12-04T05:37:57.271 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up librbd1 (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:57.332 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libcephfs2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:57.381 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-mgr-modules-core (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:57.458 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libradosstriper1 (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:57.490 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-fuse (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:57.575 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-12-04T05:37:57.689 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-04T05:37:57.689 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-04T05:37:57.843 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up librbd1 (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:57.969 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mgr-modules-core (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:58.121 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-fuse (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:58.233 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libcephfs-dev (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:58.334 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-04T05:37:58.334 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-04T05:37:58.359 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-rados (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:58.485 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up librgw2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:58.611 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up cephfs-top (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:58.721 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-04T05:37:58.871 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libcephfs-dev (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:58.973 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-rados (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:59.073 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up librgw2 (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:59.199 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up cephfs-top (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:37:59.317 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-04T05:37:59.941 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-rbd (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:00.067 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up rbd-fuse (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:00.193 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-rgw (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:00.319 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-cephfs (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:00.445 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up cephfs-shell (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:00.514 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-rbd (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:00.554 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-common (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:00.615 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up rbd-fuse (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:00.724 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-rgw (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:00.850 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-cephfs (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:00.976 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up cephfs-shell (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:01.064 INFO:teuthology.orchestra.run.smithi195.stdout:Adding group ceph....done 2023-12-04T05:38:01.102 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-common (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:01.424 INFO:teuthology.orchestra.run.smithi195.stdout:Adding system user ceph....done 2023-12-04T05:38:01.617 INFO:teuthology.orchestra.run.smithi195.stdout:Setting system user ceph properties....done 2023-12-04T05:38:01.630 INFO:teuthology.orchestra.run.smithi195.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-12-04T05:38:01.653 INFO:teuthology.orchestra.run.smithi107.stdout:Adding group ceph....done 2023-12-04T05:38:01.769 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-12-04T05:38:01.956 INFO:teuthology.orchestra.run.smithi107.stdout:Adding system user ceph....done 2023-12-04T05:38:02.090 INFO:teuthology.orchestra.run.smithi107.stdout:Setting system user ceph properties....done 2023-12-04T05:38:02.106 INFO:teuthology.orchestra.run.smithi107.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-12-04T05:38:02.138 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-12-04T05:38:02.244 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-12-04T05:38:02.632 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-12-04T05:38:02.688 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up cephfs-mirror (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:02.889 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-12-04T05:38:02.889 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-12-04T05:38:03.170 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up cephfs-mirror (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:03.383 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-12-04T05:38:03.383 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-12-04T05:38:03.481 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-test (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:03.607 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up radosgw (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:03.954 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-test (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:04.080 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up radosgw (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:04.106 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-04T05:38:04.106 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-04T05:38:04.582 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-04T05:38:04.582 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-04T05:38:04.659 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-base (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:05.006 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-12-04T05:38:05.090 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-base (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:05.445 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-12-04T05:38:05.544 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-mds (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:05.760 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-04T05:38:05.761 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-04T05:38:05.992 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mds (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:06.196 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-04T05:38:06.196 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-04T05:38:06.254 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-mgr (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:06.470 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-04T05:38:06.471 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-04T05:38:06.845 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mgr (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:06.980 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-osd (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:07.065 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-04T05:38:07.065 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-04T05:38:07.290 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-04T05:38:07.290 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-04T05:38:07.604 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-osd (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:07.833 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-mgr-k8sevents (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:07.929 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-04T05:38:07.929 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-04T05:38:07.959 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:08.110 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-mon (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:08.326 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-04T05:38:08.326 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-04T05:38:08.502 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mgr-k8sevents (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:08.694 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:08.878 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-mgr-cephadm (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:09.029 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:09.072 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mon (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:09.154 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-mgr-dashboard (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:09.297 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-volume (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:09.351 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-04T05:38:09.351 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-04T05:38:09.642 INFO:teuthology.orchestra.run.smithi195.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2023-12-04T05:38:09.743 INFO:teuthology.orchestra.run.smithi195.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-12-04T05:38:09.895 INFO:teuthology.orchestra.run.smithi195.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-12-04T05:38:09.910 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mgr-cephadm (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:10.061 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:10.196 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mgr-dashboard (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:10.339 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-volume (18.0.0-7531-g47c1c547-1jammy) ... 2023-12-04T05:38:10.674 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2023-12-04T05:38:10.767 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-12-04T05:38:10.894 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-12-04T05:38:11.012 INFO:teuthology.orchestra.run.smithi195.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-12-04T05:38:11.643 INFO:teuthology.orchestra.run.smithi195.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-04T05:38:11.646 DEBUG:teuthology.orchestra.run.smithi195:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-12-04T05:38:11.669 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-12-04T05:38:11.699 INFO:teuthology.orchestra.run.smithi195.stdout:Reading package lists... 2023-12-04T05:38:11.879 INFO:teuthology.orchestra.run.smithi195.stdout:Building dependency tree... 2023-12-04T05:38:11.879 INFO:teuthology.orchestra.run.smithi195.stdout:Reading state information... 2023-12-04T05:38:12.027 INFO:teuthology.orchestra.run.smithi195.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-12-04T05:38:12.027 INFO:teuthology.orchestra.run.smithi195.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-12-04T05:38:12.027 INFO:teuthology.orchestra.run.smithi195.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-12-04T05:38:12.027 INFO:teuthology.orchestra.run.smithi195.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-12-04T05:38:12.027 INFO:teuthology.orchestra.run.smithi195.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-12-04T05:38:12.027 INFO:teuthology.orchestra.run.smithi195.stdout:The following packages were automatically installed and are no longer required: 2023-12-04T05:38:12.027 INFO:teuthology.orchestra.run.smithi195.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-04T05:38:12.028 INFO:teuthology.orchestra.run.smithi195.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-04T05:38:12.028 INFO:teuthology.orchestra.run.smithi195.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-04T05:38:12.028 INFO:teuthology.orchestra.run.smithi195.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-04T05:38:12.028 INFO:teuthology.orchestra.run.smithi195.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-04T05:38:12.028 INFO:teuthology.orchestra.run.smithi195.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-04T05:38:12.028 INFO:teuthology.orchestra.run.smithi195.stdout:The following additional packages will be installed: 2023-12-04T05:38:12.029 INFO:teuthology.orchestra.run.smithi195.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-12-04T05:38:12.029 INFO:teuthology.orchestra.run.smithi195.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-12-04T05:38:12.029 INFO:teuthology.orchestra.run.smithi195.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-12-04T05:38:12.029 INFO:teuthology.orchestra.run.smithi195.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-12-04T05:38:12.029 INFO:teuthology.orchestra.run.smithi195.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-12-04T05:38:12.030 INFO:teuthology.orchestra.run.smithi195.stdout:Suggested packages: 2023-12-04T05:38:12.030 INFO:teuthology.orchestra.run.smithi195.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-12-04T05:38:12.030 INFO:teuthology.orchestra.run.smithi195.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-12-04T05:38:12.030 INFO:teuthology.orchestra.run.smithi195.stdout: libengine-pkcs11-openssl 2023-12-04T05:38:12.067 INFO:teuthology.orchestra.run.smithi195.stdout:The following NEW packages will be installed: 2023-12-04T05:38:12.067 INFO:teuthology.orchestra.run.smithi195.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-12-04T05:38:12.067 INFO:teuthology.orchestra.run.smithi195.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-12-04T05:38:12.067 INFO:teuthology.orchestra.run.smithi195.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-12-04T05:38:12.067 INFO:teuthology.orchestra.run.smithi195.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-12-04T05:38:12.067 INFO:teuthology.orchestra.run.smithi195.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-12-04T05:38:12.068 INFO:teuthology.orchestra.run.smithi195.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-12-04T05:38:12.068 INFO:teuthology.orchestra.run.smithi195.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-12-04T05:38:12.068 INFO:teuthology.orchestra.run.smithi195.stdout:The following packages will be upgraded: 2023-12-04T05:38:12.069 INFO:teuthology.orchestra.run.smithi195.stdout: libssl3 2023-12-04T05:38:12.319 INFO:teuthology.orchestra.run.smithi107.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-04T05:38:12.322 DEBUG:teuthology.orchestra.run.smithi107:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-12-04T05:38:12.375 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2023-12-04T05:38:12.566 INFO:teuthology.orchestra.run.smithi107.stdout:Building dependency tree... 2023-12-04T05:38:12.566 INFO:teuthology.orchestra.run.smithi107.stdout:Reading state information... 2023-12-04T05:38:12.728 INFO:teuthology.orchestra.run.smithi107.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-12-04T05:38:12.728 INFO:teuthology.orchestra.run.smithi107.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-12-04T05:38:12.729 INFO:teuthology.orchestra.run.smithi107.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-12-04T05:38:12.729 INFO:teuthology.orchestra.run.smithi107.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-12-04T05:38:12.729 INFO:teuthology.orchestra.run.smithi107.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-12-04T05:38:12.729 INFO:teuthology.orchestra.run.smithi107.stdout:The following packages were automatically installed and are no longer required: 2023-12-04T05:38:12.729 INFO:teuthology.orchestra.run.smithi107.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-04T05:38:12.729 INFO:teuthology.orchestra.run.smithi107.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-04T05:38:12.729 INFO:teuthology.orchestra.run.smithi107.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-04T05:38:12.729 INFO:teuthology.orchestra.run.smithi107.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-04T05:38:12.730 INFO:teuthology.orchestra.run.smithi107.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-04T05:38:12.730 INFO:teuthology.orchestra.run.smithi107.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-04T05:38:12.730 INFO:teuthology.orchestra.run.smithi107.stdout:The following additional packages will be installed: 2023-12-04T05:38:12.730 INFO:teuthology.orchestra.run.smithi107.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-12-04T05:38:12.730 INFO:teuthology.orchestra.run.smithi107.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-12-04T05:38:12.731 INFO:teuthology.orchestra.run.smithi107.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-12-04T05:38:12.731 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-12-04T05:38:12.731 INFO:teuthology.orchestra.run.smithi107.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-12-04T05:38:12.731 INFO:teuthology.orchestra.run.smithi107.stdout:Suggested packages: 2023-12-04T05:38:12.731 INFO:teuthology.orchestra.run.smithi107.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-12-04T05:38:12.731 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-12-04T05:38:12.732 INFO:teuthology.orchestra.run.smithi107.stdout: libengine-pkcs11-openssl 2023-12-04T05:38:12.772 INFO:teuthology.orchestra.run.smithi107.stdout:The following NEW packages will be installed: 2023-12-04T05:38:12.772 INFO:teuthology.orchestra.run.smithi107.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-12-04T05:38:12.772 INFO:teuthology.orchestra.run.smithi107.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-12-04T05:38:12.772 INFO:teuthology.orchestra.run.smithi107.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-12-04T05:38:12.772 INFO:teuthology.orchestra.run.smithi107.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-12-04T05:38:12.773 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-12-04T05:38:12.773 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-12-04T05:38:12.773 INFO:teuthology.orchestra.run.smithi107.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-12-04T05:38:12.773 INFO:teuthology.orchestra.run.smithi107.stdout:The following packages will be upgraded: 2023-12-04T05:38:12.773 INFO:teuthology.orchestra.run.smithi107.stdout: libssl3 2023-12-04T05:38:12.826 INFO:teuthology.orchestra.run.smithi107.stdout:1 upgraded, 39 newly installed, 0 to remove and 261 not upgraded. 2023-12-04T05:38:12.827 INFO:teuthology.orchestra.run.smithi107.stdout:Need to get 54.6 MB of archives. 2023-12-04T05:38:12.827 INFO:teuthology.orchestra.run.smithi107.stdout:After this operation, 202 MB of additional disk space will be used. 2023-12-04T05:38:12.827 INFO:teuthology.orchestra.run.smithi107.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-12-04T05:38:12.948 INFO:teuthology.orchestra.run.smithi107.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2023-12-04T05:38:13.112 INFO:teuthology.orchestra.run.smithi107.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-12-04T05:38:13.172 INFO:teuthology.orchestra.run.smithi107.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-12-04T05:38:13.174 INFO:teuthology.orchestra.run.smithi107.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-12-04T05:38:13.177 INFO:teuthology.orchestra.run.smithi107.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-12-04T05:38:13.181 INFO:teuthology.orchestra.run.smithi107.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-12-04T05:38:13.182 INFO:teuthology.orchestra.run.smithi107.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-12-04T05:38:13.183 INFO:teuthology.orchestra.run.smithi107.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-12-04T05:38:13.185 INFO:teuthology.orchestra.run.smithi107.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-12-04T05:38:13.185 INFO:teuthology.orchestra.run.smithi107.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-12-04T05:38:13.186 INFO:teuthology.orchestra.run.smithi107.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-12-04T05:38:13.206 INFO:teuthology.orchestra.run.smithi107.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-12-04T05:38:13.717 INFO:teuthology.orchestra.run.smithi107.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-12-04T05:38:13.720 INFO:teuthology.orchestra.run.smithi107.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-12-04T05:38:13.720 INFO:teuthology.orchestra.run.smithi107.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-12-04T05:38:13.725 INFO:teuthology.orchestra.run.smithi107.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-12-04T05:38:13.725 INFO:teuthology.orchestra.run.smithi107.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-12-04T05:38:13.732 INFO:teuthology.orchestra.run.smithi107.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.9-0ubuntu0.22.04.1 [141 kB] 2023-12-04T05:38:13.734 INFO:teuthology.orchestra.run.smithi107.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-12-04T05:38:13.745 INFO:teuthology.orchestra.run.smithi107.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-12-04T05:38:13.746 INFO:teuthology.orchestra.run.smithi107.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2023-12-04T05:38:13.777 INFO:teuthology.orchestra.run.smithi107.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-12-04T05:38:13.777 INFO:teuthology.orchestra.run.smithi107.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-12-04T05:38:13.788 INFO:teuthology.orchestra.run.smithi107.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-12-04T05:38:13.808 INFO:teuthology.orchestra.run.smithi107.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-12-04T05:38:13.811 INFO:teuthology.orchestra.run.smithi107.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-12-04T05:38:13.811 INFO:teuthology.orchestra.run.smithi107.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-12-04T05:38:13.811 INFO:teuthology.orchestra.run.smithi107.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-12-04T05:38:13.812 INFO:teuthology.orchestra.run.smithi107.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.9-0ubuntu0.22.04.1 [1,222 kB] 2023-12-04T05:38:13.825 INFO:teuthology.orchestra.run.smithi107.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-12-04T05:38:13.826 INFO:teuthology.orchestra.run.smithi107.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.9-0ubuntu0.22.04.1 [16.1 MB] 2023-12-04T05:38:14.116 INFO:teuthology.orchestra.run.smithi107.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-12-04T05:38:14.116 INFO:teuthology.orchestra.run.smithi107.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-12-04T05:38:14.116 INFO:teuthology.orchestra.run.smithi107.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-12-04T05:38:14.116 INFO:teuthology.orchestra.run.smithi107.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-12-04T05:38:14.117 INFO:teuthology.orchestra.run.smithi107.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-12-04T05:38:14.117 INFO:teuthology.orchestra.run.smithi107.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-12-04T05:38:14.129 INFO:teuthology.orchestra.run.smithi107.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-12-04T05:38:14.130 INFO:teuthology.orchestra.run.smithi107.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-12-04T05:38:14.533 INFO:teuthology.orchestra.run.smithi107.stdout:Preconfiguring packages ... 2023-12-04T05:38:14.671 INFO:teuthology.orchestra.run.smithi107.stdout:Fetched 54.6 MB in 1s (40.4 MB/s) 2023-12-04T05:38:14.801 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package flex. 2023-12-04T05:38:14.885 INFO:teuthology.orchestra.run.smithi107.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173619 files and directories currently installed.) 2023-12-04T05:38:14.891 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-12-04T05:38:14.933 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking flex (2.6.4-8build2) ... 2023-12-04T05:38:15.500 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2023-12-04T05:38:15.672 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2023-12-04T05:38:16.058 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2023-12-04T05:38:16.582 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package bison. 2023-12-04T05:38:16.685 INFO:teuthology.orchestra.run.smithi107.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173692 files and directories currently installed.) 2023-12-04T05:38:16.690 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-12-04T05:38:16.741 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-12-04T05:38:17.336 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-12-04T05:38:17.364 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-12-04T05:38:17.402 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-12-04T05:38:17.681 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package dump. 2023-12-04T05:38:17.708 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-12-04T05:38:17.739 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking dump (0.4b47-3) ... 2023-12-04T05:38:17.804 INFO:teuthology.orchestra.run.smithi195.stdout:1 upgraded, 39 newly installed, 0 to remove and 261 not upgraded. 2023-12-04T05:38:17.804 INFO:teuthology.orchestra.run.smithi195.stdout:Need to get 54.6 MB of archives. 2023-12-04T05:38:17.804 INFO:teuthology.orchestra.run.smithi195.stdout:After this operation, 202 MB of additional disk space will be used. 2023-12-04T05:38:17.804 INFO:teuthology.orchestra.run.smithi195.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-12-04T05:38:17.903 INFO:teuthology.orchestra.run.smithi195.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2023-12-04T05:38:17.962 INFO:teuthology.orchestra.run.smithi195.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-12-04T05:38:17.976 INFO:teuthology.orchestra.run.smithi195.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-12-04T05:38:17.977 INFO:teuthology.orchestra.run.smithi195.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-12-04T05:38:17.980 INFO:teuthology.orchestra.run.smithi195.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-12-04T05:38:17.982 INFO:teuthology.orchestra.run.smithi195.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-12-04T05:38:17.982 INFO:teuthology.orchestra.run.smithi195.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-12-04T05:38:17.985 INFO:teuthology.orchestra.run.smithi195.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-12-04T05:38:17.986 INFO:teuthology.orchestra.run.smithi195.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-12-04T05:38:17.986 INFO:teuthology.orchestra.run.smithi195.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-12-04T05:38:17.986 INFO:teuthology.orchestra.run.smithi195.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-12-04T05:38:18.000 INFO:teuthology.orchestra.run.smithi195.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-12-04T05:38:18.025 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package indent. 2023-12-04T05:38:18.053 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-12-04T05:38:18.092 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-12-04T05:38:18.311 INFO:teuthology.orchestra.run.smithi195.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-12-04T05:38:18.314 INFO:teuthology.orchestra.run.smithi195.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-12-04T05:38:18.314 INFO:teuthology.orchestra.run.smithi195.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-12-04T05:38:18.318 INFO:teuthology.orchestra.run.smithi195.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-12-04T05:38:18.318 INFO:teuthology.orchestra.run.smithi195.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-12-04T05:38:18.325 INFO:teuthology.orchestra.run.smithi195.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.9-0ubuntu0.22.04.1 [141 kB] 2023-12-04T05:38:18.327 INFO:teuthology.orchestra.run.smithi195.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-12-04T05:38:18.337 INFO:teuthology.orchestra.run.smithi195.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-12-04T05:38:18.337 INFO:teuthology.orchestra.run.smithi195.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2023-12-04T05:38:18.352 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-12-04T05:38:18.367 INFO:teuthology.orchestra.run.smithi195.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-12-04T05:38:18.367 INFO:teuthology.orchestra.run.smithi195.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-12-04T05:38:18.368 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-12-04T05:38:18.378 INFO:teuthology.orchestra.run.smithi195.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-12-04T05:38:18.398 INFO:teuthology.orchestra.run.smithi195.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-12-04T05:38:18.401 INFO:teuthology.orchestra.run.smithi195.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-12-04T05:38:18.401 INFO:teuthology.orchestra.run.smithi195.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-12-04T05:38:18.402 INFO:teuthology.orchestra.run.smithi195.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-12-04T05:38:18.402 INFO:teuthology.orchestra.run.smithi195.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.9-0ubuntu0.22.04.1 [1,222 kB] 2023-12-04T05:38:18.411 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-12-04T05:38:18.412 INFO:teuthology.orchestra.run.smithi195.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-12-04T05:38:18.413 INFO:teuthology.orchestra.run.smithi195.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.9-0ubuntu0.22.04.1 [16.1 MB] 2023-12-04T05:38:18.612 INFO:teuthology.orchestra.run.smithi195.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-12-04T05:38:18.612 INFO:teuthology.orchestra.run.smithi195.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-12-04T05:38:18.612 INFO:teuthology.orchestra.run.smithi195.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-12-04T05:38:18.612 INFO:teuthology.orchestra.run.smithi195.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-12-04T05:38:18.612 INFO:teuthology.orchestra.run.smithi195.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-12-04T05:38:18.612 INFO:teuthology.orchestra.run.smithi195.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-12-04T05:38:18.623 INFO:teuthology.orchestra.run.smithi195.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-12-04T05:38:18.624 INFO:teuthology.orchestra.run.smithi195.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-12-04T05:38:18.707 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-12-04T05:38:18.734 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-12-04T05:38:18.773 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-12-04T05:38:19.025 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-12-04T05:38:19.026 INFO:teuthology.orchestra.run.smithi195.stdout:Preconfiguring packages ... 2023-12-04T05:38:19.041 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-12-04T05:38:19.084 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-12-04T05:38:19.170 INFO:teuthology.orchestra.run.smithi195.stdout:Fetched 54.6 MB in 7s (8,336 kB/s) 2023-12-04T05:38:19.311 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package flex. 2023-12-04T05:38:19.414 INFO:teuthology.orchestra.run.smithi195.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173619 files and directories currently installed.) 2023-12-04T05:38:19.419 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-12-04T05:38:19.420 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libfl2:amd64. 2023-12-04T05:38:19.448 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-12-04T05:38:19.459 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking flex (2.6.4-8build2) ... 2023-12-04T05:38:19.487 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-12-04T05:38:19.731 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-12-04T05:38:19.747 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-12-04T05:38:19.781 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-12-04T05:38:20.008 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libjson-perl. 2023-12-04T05:38:20.017 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2023-12-04T05:38:20.024 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-12-04T05:38:20.067 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-12-04T05:38:20.174 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2023-12-04T05:38:20.378 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libllvm14:amd64. 2023-12-04T05:38:20.406 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-12-04T05:38:20.445 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-12-04T05:38:20.550 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2023-12-04T05:38:21.092 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package bison. 2023-12-04T05:38:21.196 INFO:teuthology.orchestra.run.smithi195.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173692 files and directories currently installed.) 2023-12-04T05:38:21.202 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-12-04T05:38:21.252 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-12-04T05:38:21.661 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-12-04T05:38:21.688 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-12-04T05:38:21.727 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-12-04T05:38:22.022 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package dump. 2023-12-04T05:38:22.049 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-12-04T05:38:22.073 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-12-04T05:38:22.080 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking dump (0.4b47-3) ... 2023-12-04T05:38:22.100 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-12-04T05:38:22.159 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-12-04T05:38:22.325 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package indent. 2023-12-04T05:38:22.341 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-12-04T05:38:22.383 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-12-04T05:38:22.391 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libmbim-proxy. 2023-12-04T05:38:22.416 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-12-04T05:38:22.458 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-12-04T05:38:22.678 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-12-04T05:38:22.706 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-12-04T05:38:22.761 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-12-04T05:38:22.786 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-12-04T05:38:22.815 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-12-04T05:38:22.852 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-12-04T05:38:23.090 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-12-04T05:38:23.118 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-12-04T05:38:23.156 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-12-04T05:38:23.205 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libndp0:amd64. 2023-12-04T05:38:23.234 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-12-04T05:38:23.272 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-12-04T05:38:23.384 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-12-04T05:38:23.400 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-12-04T05:38:23.437 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-12-04T05:38:23.550 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libnm0:amd64. 2023-12-04T05:38:23.577 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-12-04T05:38:23.608 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-12-04T05:38:23.762 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libfl2:amd64. 2023-12-04T05:38:23.790 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-12-04T05:38:23.829 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-12-04T05:38:23.944 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libpq5:amd64. 2023-12-04T05:38:23.971 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../16-libpq5_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-12-04T05:38:24.010 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T05:38:24.089 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-12-04T05:38:24.105 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-12-04T05:38:24.148 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-12-04T05:38:24.397 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-12-04T05:38:24.417 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libjson-perl. 2023-12-04T05:38:24.425 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-12-04T05:38:24.445 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-12-04T05:38:24.463 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-12-04T05:38:24.484 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-12-04T05:38:24.749 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libqmi-proxy. 2023-12-04T05:38:24.766 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-12-04T05:38:24.770 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libllvm14:amd64. 2023-12-04T05:38:24.798 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-12-04T05:38:24.807 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-12-04T05:38:24.836 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-12-04T05:38:25.026 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-12-04T05:38:25.042 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2023-12-04T05:38:25.076 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-12-04T05:38:25.538 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-12-04T05:38:25.566 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-12-04T05:38:25.604 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-12-04T05:38:25.957 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package modemmanager. 2023-12-04T05:38:25.985 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-12-04T05:38:26.024 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-12-04T05:38:26.439 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-12-04T05:38:26.467 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-12-04T05:38:26.497 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-12-04T05:38:26.580 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package network-manager. 2023-12-04T05:38:26.608 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-12-04T05:38:26.647 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-12-04T05:38:26.750 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libmbim-proxy. 2023-12-04T05:38:26.766 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-12-04T05:38:26.808 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-12-04T05:38:27.127 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-12-04T05:38:27.155 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-12-04T05:38:27.194 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-12-04T05:38:27.245 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ppp. 2023-12-04T05:38:27.273 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-12-04T05:38:27.497 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libndp0:amd64. 2023-12-04T05:38:27.525 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-12-04T05:38:27.554 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-12-04T05:38:27.618 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-12-04T05:38:27.892 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libnm0:amd64. 2023-12-04T05:38:27.920 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-12-04T05:38:27.951 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package pptp-linux. 2023-12-04T05:38:27.958 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-12-04T05:38:27.978 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-12-04T05:38:28.017 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-12-04T05:38:28.270 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package network-manager-pptp. 2023-12-04T05:38:28.297 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-12-04T05:38:28.328 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libpq5:amd64. 2023-12-04T05:38:28.336 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-12-04T05:38:28.355 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../16-libpq5_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-12-04T05:38:28.394 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T05:38:28.614 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package postgresql-client-common. 2023-12-04T05:38:28.642 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-12-04T05:38:28.681 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking postgresql-client-common (238) ... 2023-12-04T05:38:28.822 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-12-04T05:38:28.850 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-12-04T05:38:28.880 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-12-04T05:38:28.985 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package postgresql-client-14. 2023-12-04T05:38:29.012 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../27-postgresql-client-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-12-04T05:38:29.065 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T05:38:29.175 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libqmi-proxy. 2023-12-04T05:38:29.191 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-12-04T05:38:29.233 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-12-04T05:38:29.477 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-12-04T05:38:29.504 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2023-12-04T05:38:29.543 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-12-04T05:38:29.656 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package postgresql-common. 2023-12-04T05:38:29.684 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-12-04T05:38:29.741 INFO:teuthology.orchestra.run.smithi107.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-12-04T05:38:29.770 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking postgresql-common (238) ... 2023-12-04T05:38:29.997 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-12-04T05:38:30.028 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-12-04T05:38:30.063 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-12-04T05:38:30.222 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package postgresql-14. 2023-12-04T05:38:30.250 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../29-postgresql-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-12-04T05:38:30.292 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T05:38:30.400 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package modemmanager. 2023-12-04T05:38:30.428 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-12-04T05:38:30.466 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-12-04T05:38:31.080 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package network-manager. 2023-12-04T05:38:31.108 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-12-04T05:38:31.155 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-12-04T05:38:31.691 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package postgresql. 2023-12-04T05:38:31.707 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-12-04T05:38:31.749 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking postgresql (14+238) ... 2023-12-04T05:38:31.821 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ppp. 2023-12-04T05:38:31.849 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-12-04T05:38:31.951 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package postgresql-client. 2023-12-04T05:38:31.979 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-12-04T05:38:32.018 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking postgresql-client (14+238) ... 2023-12-04T05:38:32.165 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-12-04T05:38:32.270 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package postgresql-contrib. 2023-12-04T05:38:32.286 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-12-04T05:38:32.328 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking postgresql-contrib (14+238) ... 2023-12-04T05:38:32.475 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package pptp-linux. 2023-12-04T05:38:32.504 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-12-04T05:38:32.542 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-12-04T05:38:32.758 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package tcl8.6. 2023-12-04T05:38:32.778 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package network-manager-pptp. 2023-12-04T05:38:32.786 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-12-04T05:38:32.794 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-12-04T05:38:32.824 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-12-04T05:38:32.827 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-12-04T05:38:33.102 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package tcl. 2023-12-04T05:38:33.130 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-12-04T05:38:33.173 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-12-04T05:38:33.530 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package wpasupplicant. 2023-12-04T05:38:33.558 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-12-04T05:38:33.590 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package postgresql-client-common. 2023-12-04T05:38:33.596 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-12-04T05:38:33.618 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-12-04T05:38:33.657 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking postgresql-client-common (238) ... 2023-12-04T05:38:33.942 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-12-04T05:38:33.944 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package postgresql-client-14. 2023-12-04T05:38:33.959 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-12-04T05:38:33.972 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../27-postgresql-client-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-12-04T05:38:33.992 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-12-04T05:38:34.010 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T05:38:34.337 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package usb-modeswitch. 2023-12-04T05:38:34.365 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-12-04T05:38:34.403 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-12-04T05:38:34.582 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package postgresql-common. 2023-12-04T05:38:34.610 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-12-04T05:38:34.675 INFO:teuthology.orchestra.run.smithi195.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-12-04T05:38:34.683 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up postgresql-client-common (238) ... 2023-12-04T05:38:34.704 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking postgresql-common (238) ... 2023-12-04T05:38:34.884 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up flex (2.6.4-8build2) ... 2023-12-04T05:38:35.044 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-12-04T05:38:35.080 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package postgresql-14. 2023-12-04T05:38:35.108 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../29-postgresql-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-12-04T05:38:35.161 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T05:38:35.170 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T05:38:35.288 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-12-04T05:38:35.389 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-12-04T05:38:35.498 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T05:38:36.002 INFO:teuthology.orchestra.run.smithi107.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-12-04T05:38:36.078 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-12-04T05:38:36.410 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package postgresql. 2023-12-04T05:38:36.426 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-12-04T05:38:36.426 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-12-04T05:38:36.432 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-12-04T05:38:36.477 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking postgresql (14+238) ... 2023-12-04T05:38:36.679 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package postgresql-client. 2023-12-04T05:38:36.695 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-12-04T05:38:36.729 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking postgresql-client (14+238) ... 2023-12-04T05:38:36.898 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-12-04T05:38:36.948 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package postgresql-contrib. 2023-12-04T05:38:36.964 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-12-04T05:38:37.006 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking postgresql-contrib (14+238) ... 2023-12-04T05:38:37.024 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-12-04T05:38:37.150 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-12-04T05:38:37.276 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-12-04T05:38:37.325 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package tcl8.6. 2023-12-04T05:38:37.354 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-12-04T05:38:37.393 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-12-04T05:38:37.402 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-12-04T05:38:37.661 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package tcl. 2023-12-04T05:38:37.689 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-12-04T05:38:37.701 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-12-04T05:38:37.732 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-12-04T05:38:38.014 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package wpasupplicant. 2023-12-04T05:38:38.030 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-12-04T05:38:38.062 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-12-04T05:38:38.072 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-12-04T05:38:38.443 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-12-04T05:38:38.460 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-12-04T05:38:38.501 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-12-04T05:38:38.676 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-12-04T05:38:38.877 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up dump (0.4b47-3) ... 2023-12-04T05:38:38.904 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package usb-modeswitch. 2023-12-04T05:38:38.933 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-12-04T05:38:38.980 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-12-04T05:38:39.004 INFO:teuthology.orchestra.run.smithi107.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-12-04T05:38:39.078 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-12-04T05:38:39.196 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-12-04T05:38:39.259 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up postgresql-client-common (238) ... 2023-12-04T05:38:39.297 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up tcl (8.6.11+1build2) ... 2023-12-04T05:38:39.398 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libjson-perl (4.04000-1) ... 2023-12-04T05:38:39.469 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up flex (2.6.4-8build2) ... 2023-12-04T05:38:39.524 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-12-04T05:38:39.586 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-12-04T05:38:39.614 INFO:teuthology.orchestra.run.smithi107.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-12-04T05:38:39.688 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T05:38:39.692 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-12-04T05:38:39.788 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-12-04T05:38:39.818 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-12-04T05:38:39.914 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-12-04T05:38:39.995 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-12-04T05:38:40.040 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T05:38:40.121 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-12-04T05:38:40.222 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-12-04T05:38:40.356 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up postgresql-client (14+238) ... 2023-12-04T05:38:40.474 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-12-04T05:38:40.556 INFO:teuthology.orchestra.run.smithi195.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-12-04T05:38:40.629 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-12-04T05:38:40.978 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:38:40.978 INFO:teuthology.orchestra.run.smithi107.stdout:The following network interfaces were found in /etc/network/interfaces 2023-12-04T05:38:40.978 INFO:teuthology.orchestra.run.smithi107.stdout:which means they are currently configured by ifupdown: 2023-12-04T05:38:40.979 INFO:teuthology.orchestra.run.smithi107.stdout:- enp3s0f1 2023-12-04T05:38:40.979 INFO:teuthology.orchestra.run.smithi107.stdout:If you want to manage those interfaces with NetworkManager instead 2023-12-04T05:38:40.979 INFO:teuthology.orchestra.run.smithi107.stdout:remove their configuration from /etc/network/interfaces. 2023-12-04T05:38:40.979 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:38:41.024 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-12-04T05:38:41.024 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-12-04T05:38:41.094 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-12-04T05:38:41.444 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-12-04T05:38:41.574 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-12-04T05:38:41.675 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-12-04T05:38:41.760 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-12-04T05:38:41.792 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-12-04T05:38:41.918 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-12-04T05:38:42.044 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-12-04T05:38:42.170 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-12-04T05:38:42.296 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-12-04T05:38:42.301 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-12-04T05:38:42.444 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-12-04T05:38:42.573 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-12-04T05:38:42.682 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up postgresql-common (238) ... 2023-12-04T05:38:42.927 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-12-04T05:38:43.053 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up dump (0.4b47-3) ... 2023-12-04T05:38:43.148 INFO:teuthology.orchestra.run.smithi195.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-12-04T05:38:43.212 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-12-04T05:38:43.338 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-12-04T05:38:43.464 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up tcl (8.6.11+1build2) ... 2023-12-04T05:38:43.666 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libjson-perl (4.04000-1) ... 2023-12-04T05:38:43.718 INFO:teuthology.orchestra.run.smithi107.stdout:Adding user postgres to group ssl-cert 2023-12-04T05:38:43.956 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:38:43.956 INFO:teuthology.orchestra.run.smithi107.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-12-04T05:38:44.033 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-12-04T05:38:44.148 INFO:teuthology.orchestra.run.smithi107.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-12-04T05:38:44.148 INFO:teuthology.orchestra.run.smithi107.stdout:Removing obsolete dictionary files: 2023-12-04T05:38:44.149 INFO:teuthology.orchestra.run.smithi195.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-12-04T05:38:44.227 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-12-04T05:38:44.353 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-12-04T05:38:44.529 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-12-04T05:38:44.655 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-12-04T05:38:44.768 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-12-04T05:38:44.781 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-12-04T05:38:44.949 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up postgresql-client (14+238) ... 2023-12-04T05:38:45.066 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-12-04T05:38:45.217 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T05:38:45.548 INFO:teuthology.orchestra.run.smithi107.stdout:Creating new PostgreSQL cluster 14/main ... 2023-12-04T05:38:45.552 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:38:45.552 INFO:teuthology.orchestra.run.smithi195.stdout:The following network interfaces were found in /etc/network/interfaces 2023-12-04T05:38:45.552 INFO:teuthology.orchestra.run.smithi195.stdout:which means they are currently configured by ifupdown: 2023-12-04T05:38:45.552 INFO:teuthology.orchestra.run.smithi195.stdout:- enp3s0f1 2023-12-04T05:38:45.552 INFO:teuthology.orchestra.run.smithi195.stdout:If you want to manage those interfaces with NetworkManager instead 2023-12-04T05:38:45.552 INFO:teuthology.orchestra.run.smithi195.stdout:remove their configuration from /etc/network/interfaces. 2023-12-04T05:38:45.552 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:38:45.572 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-12-04T05:38:45.617 INFO:teuthology.orchestra.run.smithi107.stdout:The files belonging to this database system will be owned by user "postgres". 2023-12-04T05:38:45.617 INFO:teuthology.orchestra.run.smithi107.stdout:This user must also own the server process. 2023-12-04T05:38:45.618 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:38:45.618 INFO:teuthology.orchestra.run.smithi107.stdout:The database cluster will be initialized with locale "en_US". 2023-12-04T05:38:45.618 INFO:teuthology.orchestra.run.smithi107.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-12-04T05:38:45.618 INFO:teuthology.orchestra.run.smithi107.stdout:The default text search configuration will be set to "english". 2023-12-04T05:38:45.618 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:38:45.618 INFO:teuthology.orchestra.run.smithi107.stdout:Data page checksums are disabled. 2023-12-04T05:38:45.618 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:38:45.618 INFO:teuthology.orchestra.run.smithi107.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-12-04T05:38:45.618 INFO:teuthology.orchestra.run.smithi107.stdout:creating subdirectories ... ok 2023-12-04T05:38:45.618 INFO:teuthology.orchestra.run.smithi107.stdout:selecting dynamic shared memory implementation ... posix 2023-12-04T05:38:45.630 INFO:teuthology.orchestra.run.smithi107.stdout:selecting default max_connections ... 100 2023-12-04T05:38:45.648 INFO:teuthology.orchestra.run.smithi107.stdout:selecting default shared_buffers ... 128MB 2023-12-04T05:38:45.664 INFO:teuthology.orchestra.run.smithi107.stdout:selecting default time zone ... Etc/UTC 2023-12-04T05:38:45.665 INFO:teuthology.orchestra.run.smithi107.stdout:creating configuration files ... ok 2023-12-04T05:38:45.669 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-12-04T05:38:45.781 INFO:teuthology.orchestra.run.smithi107.stdout:running bootstrap script ... ok 2023-12-04T05:38:46.040 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-12-04T05:38:46.388 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-12-04T05:38:46.548 INFO:teuthology.orchestra.run.smithi107.stdout:performing post-bootstrap initialization ... ok 2023-12-04T05:38:47.011 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-12-04T05:38:47.145 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-12-04T05:38:47.220 INFO:teuthology.orchestra.run.smithi107.stdout:syncing data to disk ... ok 2023-12-04T05:38:47.271 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-12-04T05:38:47.397 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up postgresql-common (238) ... 2023-12-04T05:38:47.649 INFO:teuthology.orchestra.run.smithi107.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-12-04T05:38:48.228 INFO:teuthology.orchestra.run.smithi195.stdout:Adding user postgres to group ssl-cert 2023-12-04T05:38:48.432 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:38:48.432 INFO:teuthology.orchestra.run.smithi195.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-12-04T05:38:48.627 INFO:teuthology.orchestra.run.smithi195.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-12-04T05:38:48.628 INFO:teuthology.orchestra.run.smithi195.stdout:Removing obsolete dictionary files: 2023-12-04T05:38:49.271 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-12-04T05:38:49.707 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T05:38:50.028 INFO:teuthology.orchestra.run.smithi195.stdout:Creating new PostgreSQL cluster 14/main ... 2023-12-04T05:38:50.042 INFO:teuthology.orchestra.run.smithi195.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-12-04T05:38:50.080 INFO:teuthology.orchestra.run.smithi195.stdout:The files belonging to this database system will be owned by user "postgres". 2023-12-04T05:38:50.081 INFO:teuthology.orchestra.run.smithi195.stdout:This user must also own the server process. 2023-12-04T05:38:50.081 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:38:50.081 INFO:teuthology.orchestra.run.smithi195.stdout:The database cluster will be initialized with locale "en_US". 2023-12-04T05:38:50.081 INFO:teuthology.orchestra.run.smithi195.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-12-04T05:38:50.081 INFO:teuthology.orchestra.run.smithi195.stdout:The default text search configuration will be set to "english". 2023-12-04T05:38:50.081 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:38:50.081 INFO:teuthology.orchestra.run.smithi195.stdout:Data page checksums are disabled. 2023-12-04T05:38:50.081 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:38:50.081 INFO:teuthology.orchestra.run.smithi195.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-12-04T05:38:50.081 INFO:teuthology.orchestra.run.smithi195.stdout:creating subdirectories ... ok 2023-12-04T05:38:50.081 INFO:teuthology.orchestra.run.smithi195.stdout:selecting dynamic shared memory implementation ... posix 2023-12-04T05:38:50.094 INFO:teuthology.orchestra.run.smithi195.stdout:selecting default max_connections ... 100 2023-12-04T05:38:50.114 INFO:teuthology.orchestra.run.smithi195.stdout:selecting default shared_buffers ... 128MB 2023-12-04T05:38:50.124 INFO:teuthology.orchestra.run.smithi195.stdout:selecting default time zone ... Etc/UTC 2023-12-04T05:38:50.125 INFO:teuthology.orchestra.run.smithi195.stdout:creating configuration files ... ok 2023-12-04T05:38:50.238 INFO:teuthology.orchestra.run.smithi195.stdout:running bootstrap script ... ok 2023-12-04T05:38:50.489 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-12-04T05:38:50.599 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up postgresql-contrib (14+238) ... 2023-12-04T05:38:50.725 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up postgresql (14+238) ... 2023-12-04T05:38:50.851 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-12-04T05:38:50.896 INFO:teuthology.orchestra.run.smithi195.stdout:performing post-bootstrap initialization ... ok 2023-12-04T05:38:50.977 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-12-04T05:38:51.232 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-12-04T05:38:51.233 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-12-04T05:38:51.668 INFO:teuthology.orchestra.run.smithi195.stdout:syncing data to disk ... ok 2023-12-04T05:38:51.871 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-12-04T05:38:52.093 INFO:teuthology.orchestra.run.smithi195.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-12-04T05:38:52.173 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-12-04T05:38:52.333 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-12-04T05:38:53.686 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-12-04T05:38:53.786 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-12-04T05:38:54.270 INFO:teuthology.orchestra.run.smithi107.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-04T05:38:54.273 DEBUG:teuthology.parallel:result is None 2023-12-04T05:38:54.939 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-12-04T05:38:55.065 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up postgresql-contrib (14+238) ... 2023-12-04T05:38:55.191 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up postgresql (14+238) ... 2023-12-04T05:38:55.317 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-12-04T05:38:55.426 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-12-04T05:38:55.654 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-12-04T05:38:55.654 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-12-04T05:38:56.270 INFO:teuthology.orchestra.run.smithi195.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-12-04T05:38:56.606 INFO:teuthology.orchestra.run.smithi195.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-12-04T05:38:56.782 INFO:teuthology.orchestra.run.smithi195.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-12-04T05:38:58.027 INFO:teuthology.orchestra.run.smithi195.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-12-04T05:38:58.119 INFO:teuthology.orchestra.run.smithi195.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-12-04T05:38:58.553 INFO:teuthology.orchestra.run.smithi195.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-04T05:38:58.555 DEBUG:teuthology.parallel:result is None 2023-12-04T05:38:58.556 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=47c1c547372ed5bff7984647dc200bac0e019caf 2023-12-04T05:38:58.701 DEBUG:teuthology.orchestra.run.smithi107:> dpkg-query -W -f '${Version}' ceph 2023-12-04T05:38:58.721 INFO:teuthology.orchestra.run.smithi107.stdout:18.0.0-7531-g47c1c547-1jammy 2023-12-04T05:38:58.721 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7531-g47c1c547-1jammy 2023-12-04T05:38:58.721 INFO:teuthology.task.install:The correct ceph version 18.0.0-7531-g47c1c547-1jammy is installed. 2023-12-04T05:38:58.722 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=47c1c547372ed5bff7984647dc200bac0e019caf 2023-12-04T05:38:58.862 DEBUG:teuthology.orchestra.run.smithi195:> dpkg-query -W -f '${Version}' ceph 2023-12-04T05:38:58.881 INFO:teuthology.orchestra.run.smithi195.stdout:18.0.0-7531-g47c1c547-1jammy 2023-12-04T05:38:58.881 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7531-g47c1c547-1jammy 2023-12-04T05:38:58.881 INFO:teuthology.task.install:The correct ceph version 18.0.0-7531-g47c1c547-1jammy is installed. 2023-12-04T05:38:58.882 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-04T05:38:58.882 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:38:58.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-04T05:38:58.896 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:38:58.896 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-04T05:38:58.935 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-04T05:38:58.935 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:38:58.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/daemon-helper 2023-12-04T05:38:58.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-04T05:38:59.007 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:38:59.007 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/daemon-helper 2023-12-04T05:38:59.020 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-04T05:38:59.073 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-04T05:38:59.074 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:38:59.074 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-04T05:38:59.087 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-04T05:38:59.143 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:38:59.143 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-04T05:38:59.156 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-04T05:38:59.210 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-04T05:38:59.210 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:38:59.210 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/stdin-killer 2023-12-04T05:38:59.224 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-04T05:38:59.278 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:38:59.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/stdin-killer 2023-12-04T05:38:59.291 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-04T05:38:59.346 INFO:teuthology.run_tasks:Running task ceph... 2023-12-04T05:38:59.412 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-04T05:38:59.413 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 777 /var/log/ceph 2023-12-04T05:38:59.415 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 777 /var/log/ceph 2023-12-04T05:38:59.427 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-04T05:38:59.427 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-04T05:38:59.476 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-04T05:38:59.488 INFO:tasks.ceph:Creating extra log directories... 2023-12-04T05:38:59.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-04T05:38:59.528 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-04T05:38:59.549 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-12-04T05:38:59.550 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-12-04T05:38:59.550 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-12-04T05:38:59.550 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:38:59.550 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-12-04T05:38:59.587 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-12-04T05:38:59.662 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:38:59.662 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-12-04T05:38:59.668 DEBUG:teuthology.orchestra.run.smithi195:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-12-04T05:38:59.747 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-04T05:38:59.747 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-04T05:38:59.748 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2023-12-04_05:06:51-fs-wip-vshankar-testing-20231127.102654-distro-default-smithi/7476686', 'branch': 'wip-vshankar-testing-20231127.102654', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '7476686', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'vshankar-2023-12-04_05:06:51-fs-wip-vshankar-testing-20231127.102654-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing-20231127.102654'}, 'ceph': {'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing-20231127.102654', 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 1098, 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf', 'sleep_before_teardown': 0, 'subset': '854/1024', 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing-20231127.102654', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '47c1c547372ed5bff7984647dc200bac0e019caf', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi107.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ+YUJ6V8Zs/mK0H0UauHpLsCO+FgY8+C6vlLV1lZ8ShROidb5PMfa1jTYT/99lunhuwhLIQEe6ftNQDd9GOH0I=', 'smithi195.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFxdzf5XteYu1uo6SobO6K4ofYoKZF5RPBVzpjxQRrDSb/joT32TSkCfOjhGbd6T6OSXYnSWiFlLbP63Ry7LmfM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['suites/ffsb.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-04_05:06:51', 'tube': 'smithi', 'use_shaman': True, 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-04T05:38:59.748 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-04T05:38:59.751 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-04T05:38:59.795 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/run/ceph 2023-12-04T05:38:59.797 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m0777 -- /var/run/ceph 2023-12-04T05:38:59.851 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:38:59.851 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2023-12-04T05:38:59.856 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-04T05:38:59.856 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_1 2023-12-04T05:38:59.903 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-04T05:38:59.903 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T05:38:59.903 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 878 Links: 1 2023-12-04T05:38:59.904 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T05:38:59.904 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-12-04 05:32:28.876544457 +0000 2023-12-04T05:38:59.904 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-12-04 05:32:27.604555821 +0000 2023-12-04T05:38:59.904 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-12-04 05:32:27.604555821 +0000 2023-12-04T05:38:59.904 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2023-12-04T05:38:59.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-04T05:38:59.962 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-12-04T05:38:59.962 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-12-04T05:38:59.963 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000433012 s, 1.2 MB/s 2023-12-04T05:38:59.964 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-04T05:39:00.013 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_2 2023-12-04T05:39:00.059 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-04T05:39:00.060 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T05:39:00.060 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 901 Links: 1 2023-12-04T05:39:00.060 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T05:39:00.060 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-12-04 05:32:30.260532093 +0000 2023-12-04T05:39:00.060 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-12-04 05:32:29.112542349 +0000 2023-12-04T05:39:00.060 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-12-04 05:32:29.112542349 +0000 2023-12-04T05:39:00.060 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2023-12-04T05:39:00.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-04T05:39:00.113 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-12-04T05:39:00.114 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-12-04T05:39:00.114 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000454963 s, 1.1 MB/s 2023-12-04T05:39:00.114 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-04T05:39:00.164 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_3 2023-12-04T05:39:00.211 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-04T05:39:00.211 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T05:39:00.211 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 920 Links: 1 2023-12-04T05:39:00.211 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T05:39:00.211 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-12-04 05:32:31.672519483 +0000 2023-12-04T05:39:00.211 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-12-04 05:32:30.508529877 +0000 2023-12-04T05:39:00.211 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-12-04 05:32:30.508529877 +0000 2023-12-04T05:39:00.211 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2023-12-04T05:39:00.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-04T05:39:00.266 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-12-04T05:39:00.266 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-12-04T05:39:00.266 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.00037512 s, 1.4 MB/s 2023-12-04T05:39:00.267 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-04T05:39:00.316 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_4 2023-12-04T05:39:00.363 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-04T05:39:00.363 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T05:39:00.363 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 934 Links: 1 2023-12-04T05:39:00.363 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T05:39:00.363 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-12-04 05:32:33.056507121 +0000 2023-12-04T05:39:00.363 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-12-04 05:32:31.912517339 +0000 2023-12-04T05:39:00.363 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-12-04 05:32:31.912517339 +0000 2023-12-04T05:39:00.363 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2023-12-04T05:39:00.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-04T05:39:00.418 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-12-04T05:39:00.418 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-12-04T05:39:00.418 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000414513 s, 1.2 MB/s 2023-12-04T05:39:00.419 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-04T05:39:00.468 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-04T05:39:00.468 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:39:00.468 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/scratch_devs of=/dev/stdout 2023-12-04T05:39:00.474 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-04T05:39:00.474 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_1 2023-12-04T05:39:00.523 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-04T05:39:00.523 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T05:39:00.523 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 883 Links: 1 2023-12-04T05:39:00.523 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T05:39:00.523 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2023-12-04 05:32:32.315095292 +0000 2023-12-04T05:39:00.523 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2023-12-04 05:32:31.067110192 +0000 2023-12-04T05:39:00.523 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2023-12-04 05:32:31.067110192 +0000 2023-12-04T05:39:00.523 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2023-12-04T05:39:00.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-04T05:39:00.577 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2023-12-04T05:39:00.577 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2023-12-04T05:39:00.577 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000310238 s, 1.7 MB/s 2023-12-04T05:39:00.579 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-04T05:39:00.628 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_2 2023-12-04T05:39:00.675 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-04T05:39:00.675 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T05:39:00.675 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 897 Links: 1 2023-12-04T05:39:00.675 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T05:39:00.675 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2023-12-04 05:32:33.803077544 +0000 2023-12-04T05:39:00.675 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2023-12-04 05:32:32.619091663 +0000 2023-12-04T05:39:00.675 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2023-12-04 05:32:32.619091663 +0000 2023-12-04T05:39:00.675 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2023-12-04T05:39:00.676 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-04T05:39:00.729 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2023-12-04T05:39:00.729 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2023-12-04T05:39:00.730 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000309891 s, 1.7 MB/s 2023-12-04T05:39:00.731 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-04T05:39:00.780 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_3 2023-12-04T05:39:00.826 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-04T05:39:00.826 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T05:39:00.826 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 917 Links: 1 2023-12-04T05:39:00.827 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T05:39:00.827 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2023-12-04 05:32:35.439058066 +0000 2023-12-04T05:39:00.827 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2023-12-04 05:32:34.079074256 +0000 2023-12-04T05:39:00.827 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2023-12-04 05:32:34.079074256 +0000 2023-12-04T05:39:00.827 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2023-12-04T05:39:00.827 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-04T05:39:00.881 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2023-12-04T05:39:00.881 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2023-12-04T05:39:00.881 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.00030307 s, 1.7 MB/s 2023-12-04T05:39:00.882 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-04T05:39:00.932 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_4 2023-12-04T05:39:00.978 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-04T05:39:00.979 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T05:39:00.979 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 934 Links: 1 2023-12-04T05:39:00.979 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T05:39:00.979 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2023-12-04 05:32:36.935040263 +0000 2023-12-04T05:39:00.979 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2023-12-04 05:32:35.743054444 +0000 2023-12-04T05:39:00.979 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2023-12-04 05:32:35.743054444 +0000 2023-12-04T05:39:00.979 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2023-12-04T05:39:00.979 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-04T05:39:01.032 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2023-12-04T05:39:01.032 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2023-12-04T05:39:01.032 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.00020342 s, 2.5 MB/s 2023-12-04T05:39:01.034 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-04T05:39:01.084 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-04T05:39:01.084 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi195.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-04T05:39:01.084 INFO:tasks.ceph:Generating config... 2023-12-04T05:39:01.086 INFO:tasks.ceph:[client] client mount timeout = 600 2023-12-04T05:39:01.086 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2023-12-04T05:39:01.086 INFO:tasks.ceph:[client] debug client = 20 2023-12-04T05:39:01.087 INFO:tasks.ceph:[client] debug ms = 1 2023-12-04T05:39:01.087 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-12-04T05:39:01.087 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-12-04T05:39:01.087 INFO:tasks.ceph:[mds] debug mds = 20 2023-12-04T05:39:01.087 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-12-04T05:39:01.087 INFO:tasks.ceph:[mds] debug ms = 1 2023-12-04T05:39:01.087 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2023-12-04T05:39:01.087 INFO:tasks.ceph:[mds] mds bal merge size = 5 2023-12-04T05:39:01.087 INFO:tasks.ceph:[mds] mds bal split bits = 3 2023-12-04T05:39:01.087 INFO:tasks.ceph:[mds] mds bal split size = 100 2023-12-04T05:39:01.088 INFO:tasks.ceph:[mds] mds debug frag = True 2023-12-04T05:39:01.088 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-12-04T05:39:01.088 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2023-12-04T05:39:01.088 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-12-04T05:39:01.088 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-12-04T05:39:01.088 INFO:tasks.ceph:[mds] mds_dir_prefetch = False 2023-12-04T05:39:01.089 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2023-12-04T05:39:01.089 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2023-12-04T05:39:01.089 INFO:tasks.ceph:[mds] ms inject delay max = 1 2023-12-04T05:39:01.089 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2023-12-04T05:39:01.089 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2023-12-04T05:39:01.089 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2023-12-04T05:39:01.089 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-12-04T05:39:01.090 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-12-04T05:39:01.090 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-12-04T05:39:01.090 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-04T05:39:01.090 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-04T05:39:01.090 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-04T05:39:01.091 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-04T05:39:01.091 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-04T05:39:01.091 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-12-04T05:39:01.091 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-04T05:39:01.091 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-04T05:39:01.091 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-12-04T05:39:01.091 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-04T05:39:01.091 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-04T05:39:01.091 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-04T05:39:01.091 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-04T05:39:01.092 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-04T05:39:01.092 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-04T05:39:01.092 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-04T05:39:01.092 INFO:tasks.ceph:[osd] filestore flush min = 0 2023-12-04T05:39:01.092 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2023-12-04T05:39:01.092 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-04T05:39:01.092 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-04T05:39:01.092 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-04T05:39:01.092 INFO:tasks.ceph:[osd] ms inject delay max = 1 2023-12-04T05:39:01.093 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2023-12-04T05:39:01.093 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2023-12-04T05:39:01.093 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2023-12-04T05:39:01.093 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-12-04T05:39:01.093 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-04T05:39:01.093 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-12-04T05:39:01.093 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-04T05:39:01.093 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-12-04T05:39:01.094 INFO:tasks.ceph:Setting up mon.a... 2023-12-04T05:39:01.094 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-04T05:39:01.129 INFO:teuthology.orchestra.run.smithi107.stdout:creating /etc/ceph/ceph.keyring 2023-12-04T05:39:01.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-04T05:39:01.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-04T05:39:01.279 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.107'), ('mon.b', '172.21.15.195'), ('mon.c', '[v2:172.21.15.195:3301,v1:172.21.15.195:6790]')] 2023-12-04T05:39:01.280 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.107,172.21.15.195,[v2:172.21.15.195:3301,v1:172.21.15.195:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-12-04T05:39:01.282 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:39:01.282 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-04T05:39:01.327 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.107 --add b 172.21.15.195 --addv c '[v2:172.21.15.195:3301,v1:172.21.15.195:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-12-04T05:39:01.414 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-04T05:39:01.414 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: generated fsid 603e856b-31ac-475a-8f35-21ab7faadd48 2023-12-04T05:39:01.414 INFO:teuthology.orchestra.run.smithi107.stdout:setting min_mon_release = pacific 2023-12-04T05:39:01.414 INFO:teuthology.orchestra.run.smithi107.stdout:epoch 0 2023-12-04T05:39:01.414 INFO:teuthology.orchestra.run.smithi107.stdout:fsid 603e856b-31ac-475a-8f35-21ab7faadd48 2023-12-04T05:39:01.414 INFO:teuthology.orchestra.run.smithi107.stdout:last_changed 2023-12-04T05:39:01.415079+0000 2023-12-04T05:39:01.414 INFO:teuthology.orchestra.run.smithi107.stdout:created 2023-12-04T05:39:01.415079+0000 2023-12-04T05:39:01.414 INFO:teuthology.orchestra.run.smithi107.stdout:min_mon_release 16 (pacific) 2023-12-04T05:39:01.415 INFO:teuthology.orchestra.run.smithi107.stdout:election_strategy: 1 2023-12-04T05:39:01.415 INFO:teuthology.orchestra.run.smithi107.stdout:0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2023-12-04T05:39:01.415 INFO:teuthology.orchestra.run.smithi107.stdout:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2023-12-04T05:39:01.415 INFO:teuthology.orchestra.run.smithi107.stdout:2: [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] mon.c 2023-12-04T05:39:01.415 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-12-04T05:39:01.416 DEBUG:teuthology.orchestra.run.smithi107:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-04T05:39:01.463 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 603e856b-31ac-475a-8f35-21ab7faadd48... 2023-12-04T05:39:01.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-04T05:39:01.508 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-04T05:39:01.537 INFO:teuthology.orchestra.run.smithi107.stdout:[global] 2023-12-04T05:39:01.537 INFO:teuthology.orchestra.run.smithi107.stdout: chdir = "" 2023-12-04T05:39:01.537 INFO:teuthology.orchestra.run.smithi107.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-04T05:39:01.537 INFO:teuthology.orchestra.run.smithi107.stdout: auth supported = cephx 2023-12-04T05:39:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: filestore xattr use omap = true 2023-12-04T05:39:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: mon clock drift allowed = 1.000 2023-12-04T05:39:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: osd crush chooseleaf type = 0 2023-12-04T05:39:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: auth debug = true 2023-12-04T05:39:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on old message = true 2023-12-04T05:39:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on bug = true 2023-12-04T05:39:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-04T05:39:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: mon pg warn max object skew = 0 2023-12-04T05:39:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: # disable pg_autoscaler by default for new pools 2023-12-04T05:39:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-04T05:39:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default size = 2 2023-12-04T05:39:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow primary affinity = true 2023-12-04T05:39:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow pg remap = true 2023-12-04T05:39:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on legacy crush tunables = false 2023-12-04T05:39:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on crush straw calc version zero = false 2023-12-04T05:39:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on no sortbitwise = false 2023-12-04T05:39:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on osd down out interval zero = false 2023-12-04T05:39:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on too few osds = false 2023-12-04T05:39:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-04T05:39:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-04T05:39:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: mon_allow_pool_size_one = true 2023-12-04T05:39:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-04T05:39:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: osd default data pool replay window = 5 2023-12-04T05:39:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: mon allow pool delete = true 2023-12-04T05:39:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: mon cluster log file level = debug 2023-12-04T05:39:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: debug asserts on shutdown = true 2023-12-04T05:39:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: mon health detail to clog = false 2023-12-04T05:39:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: mon host = "172.21.15.107,172.21.15.195,[v2:172.21.15.195:3301,v1:172.21.15.195:6790]" 2023-12-04T05:39:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: fsid = 603e856b-31ac-475a-8f35-21ab7faadd48 2023-12-04T05:39:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.540 INFO:teuthology.orchestra.run.smithi107.stdout:[osd] 2023-12-04T05:39:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: osd journal size = 100 2023-12-04T05:39:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub load threshold = 5.0 2023-12-04T05:39:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub max interval = 600 2023-12-04T05:39:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: osd mclock profile = high_recovery_ops 2023-12-04T05:39:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: osd recover clone overlap = true 2023-12-04T05:39:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: osd recovery max chunk = 1048576 2023-12-04T05:39:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug shutdown = true 2023-12-04T05:39:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug op order = true 2023-12-04T05:39:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify stray on activate = true 2023-12-04T05:39:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug trim objects = true 2023-12-04T05:39:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: osd open classes on start = true 2023-12-04T05:39:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug pg log writeout = true 2023-12-04T05:39:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: osd deep scrub update digest min age = 30 2023-12-04T05:39:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: osd map max advance = 10 2023-12-04T05:39:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: journal zero on create = true 2023-12-04T05:39:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: filestore ondisk finisher threads = 3 2023-12-04T05:39:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: filestore apply finisher threads = 3 2023-12-04T05:39:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: bdev debug aio = true 2023-12-04T05:39:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug misdirected ops = true 2023-12-04T05:39:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: bdev async discard = True 2023-12-04T05:39:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: bdev enable discard = True 2023-12-04T05:39:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore allocator = bitmap 2023-12-04T05:39:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore block size = 96636764160 2023-12-04T05:39:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore fsck on mount = True 2023-12-04T05:39:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluefs = 1/20 2023-12-04T05:39:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluestore = 1/20 2023-12-04T05:39:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2023-12-04T05:39:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: debug osd = 20 2023-12-04T05:39:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: debug rocksdb = 4/10 2023-12-04T05:39:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: filestore flush min = 0 2023-12-04T05:39:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: mon client directed command retry = 5 2023-12-04T05:39:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd backfillfull_ratio = 0.85 2023-12-04T05:39:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd full ratio = 0.9 2023-12-04T05:39:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd nearfull ratio = 0.8 2023-12-04T05:39:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject delay max = 1 2023-12-04T05:39:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject delay probability = 0.005 2023-12-04T05:39:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject delay type = client mds 2023-12-04T05:39:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject socket failures = 2500 2023-12-04T05:39:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: osd blocked scrub grace period = 3600 2023-12-04T05:39:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: osd failsafe full ratio = 0.95 2023-12-04T05:39:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: osd max markdown count = 1000 2023-12-04T05:39:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: osd objectstore = bluestore 2023-12-04T05:39:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: osd op complaint time = 180 2023-12-04T05:39:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.544 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr] 2023-12-04T05:39:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2023-12-04T05:39:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: debug mgr = 20 2023-12-04T05:39:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2023-12-04T05:39:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2023-12-04T05:39:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2023-12-04T05:39:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2023-12-04T05:39:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: mgr/telemetry/nag = false 2023-12-04T05:39:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.545 INFO:teuthology.orchestra.run.smithi107.stdout:[mon] 2023-12-04T05:39:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2023-12-04T05:39:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2023-12-04T05:39:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: debug paxos = 20 2023-12-04T05:39:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2023-12-04T05:39:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: mon data avail warn = 5 2023-12-04T05:39:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: mon mgr mkfs grace = 240 2023-12-04T05:39:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2023-12-04T05:39:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd reporter subtree level = osd 2023-12-04T05:39:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd prime pg temp = true 2023-12-04T05:39:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2023-12-04T05:39:01.546 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.546 INFO:teuthology.orchestra.run.smithi107.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-04T05:39:01.546 INFO:teuthology.orchestra.run.smithi107.stdout: auth mon ticket ttl = 660 # 11m 2023-12-04T05:39:01.546 INFO:teuthology.orchestra.run.smithi107.stdout: auth service ticket ttl = 240 # 4m 2023-12-04T05:39:01.546 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.546 INFO:teuthology.orchestra.run.smithi107.stdout: # don't complain about insecure global_id in the test suite 2023-12-04T05:39:01.546 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-04T05:39:01.546 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-04T05:39:01.547 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.547 INFO:teuthology.orchestra.run.smithi107.stdout: # 1m isn't quite enough 2023-12-04T05:39:01.547 INFO:teuthology.orchestra.run.smithi107.stdout: mon_down_mkfs_grace = 2m 2023-12-04T05:39:01.547 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.547 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_filestore_osds = false 2023-12-04T05:39:01.547 INFO:teuthology.orchestra.run.smithi107.stdout: mon op complaint time = 120 2023-12-04T05:39:01.547 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:01.547 INFO:teuthology.orchestra.run.smithi107.stdout:[client] 2023-12-04T05:39:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: rgw cache enabled = true 2023-12-04T05:39:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable ops log = true 2023-12-04T05:39:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable usage log = true 2023-12-04T05:39:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-04T05:39:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-04T05:39:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: client mount timeout = 600 2023-12-04T05:39:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: client_shutdown_timeout = 120 2023-12-04T05:39:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: debug client = 20 2023-12-04T05:39:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2023-12-04T05:39:01.549 INFO:teuthology.orchestra.run.smithi107.stdout: rados mon op timeout = 900 2023-12-04T05:39:01.549 INFO:teuthology.orchestra.run.smithi107.stdout: rados osd op timeout = 900 2023-12-04T05:39:01.549 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.a] 2023-12-04T05:39:01.549 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.b] 2023-12-04T05:39:01.550 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.c] 2023-12-04T05:39:01.550 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.a] 2023-12-04T05:39:01.550 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.c] 2023-12-04T05:39:01.550 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.e] 2023-12-04T05:39:01.550 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.b] 2023-12-04T05:39:01.550 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.d] 2023-12-04T05:39:01.550 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.f] 2023-12-04T05:39:01.551 INFO:teuthology.orchestra.run.smithi107.stdout:[mds] 2023-12-04T05:39:01.551 INFO:teuthology.orchestra.run.smithi107.stdout: debug mds = 20 2023-12-04T05:39:01.551 INFO:teuthology.orchestra.run.smithi107.stdout: debug mds balancer = 20 2023-12-04T05:39:01.551 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2023-12-04T05:39:01.551 INFO:teuthology.orchestra.run.smithi107.stdout: mds bal fragment size max = 10000 2023-12-04T05:39:01.551 INFO:teuthology.orchestra.run.smithi107.stdout: mds bal merge size = 5 2023-12-04T05:39:01.551 INFO:teuthology.orchestra.run.smithi107.stdout: mds bal split bits = 3 2023-12-04T05:39:01.551 INFO:teuthology.orchestra.run.smithi107.stdout: mds bal split size = 100 2023-12-04T05:39:01.551 INFO:teuthology.orchestra.run.smithi107.stdout: mds debug frag = True 2023-12-04T05:39:01.552 INFO:teuthology.orchestra.run.smithi107.stdout: mds debug scatterstat = True 2023-12-04T05:39:01.552 INFO:teuthology.orchestra.run.smithi107.stdout: mds oft prefetch dirfrags = True 2023-12-04T05:39:01.552 INFO:teuthology.orchestra.run.smithi107.stdout: mds op complaint time = 180 2023-12-04T05:39:01.552 INFO:teuthology.orchestra.run.smithi107.stdout: mds verify scatter = True 2023-12-04T05:39:01.552 INFO:teuthology.orchestra.run.smithi107.stdout: mds_dir_prefetch = False 2023-12-04T05:39:01.552 INFO:teuthology.orchestra.run.smithi107.stdout: mds_sleep_rank_change = 5000000.0 2023-12-04T05:39:01.552 INFO:teuthology.orchestra.run.smithi107.stdout: mon client directed command retry = 5 2023-12-04T05:39:01.552 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject delay max = 1 2023-12-04T05:39:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject delay probability = 0.005 2023-12-04T05:39:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject delay type = client mds osd 2023-12-04T05:39:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject socket failures = 2500 2023-12-04T05:39:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: osd op complaint time = 180 2023-12-04T05:39:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: rados mon op timeout = 900 2023-12-04T05:39:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: rados osd op timeout = 900 2023-12-04T05:39:01.554 INFO:teuthology.orchestra.run.smithi195.stdout:[global] 2023-12-04T05:39:01.555 INFO:teuthology.orchestra.run.smithi195.stdout: chdir = "" 2023-12-04T05:39:01.555 INFO:teuthology.orchestra.run.smithi195.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-04T05:39:01.555 INFO:teuthology.orchestra.run.smithi195.stdout: auth supported = cephx 2023-12-04T05:39:01.555 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.555 INFO:teuthology.orchestra.run.smithi195.stdout: filestore xattr use omap = true 2023-12-04T05:39:01.555 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.555 INFO:teuthology.orchestra.run.smithi195.stdout: mon clock drift allowed = 1.000 2023-12-04T05:39:01.555 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.555 INFO:teuthology.orchestra.run.smithi195.stdout: osd crush chooseleaf type = 0 2023-12-04T05:39:01.555 INFO:teuthology.orchestra.run.smithi195.stdout: auth debug = true 2023-12-04T05:39:01.555 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.555 INFO:teuthology.orchestra.run.smithi195.stdout: ms die on old message = true 2023-12-04T05:39:01.555 INFO:teuthology.orchestra.run.smithi195.stdout: ms die on bug = true 2023-12-04T05:39:01.555 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.556 INFO:teuthology.orchestra.run.smithi195.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-04T05:39:01.556 INFO:teuthology.orchestra.run.smithi195.stdout: mon pg warn max object skew = 0 2023-12-04T05:39:01.556 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.556 INFO:teuthology.orchestra.run.smithi195.stdout: # disable pg_autoscaler by default for new pools 2023-12-04T05:39:01.556 INFO:teuthology.orchestra.run.smithi195.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-04T05:39:01.556 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.556 INFO:teuthology.orchestra.run.smithi195.stdout: osd pool default size = 2 2023-12-04T05:39:01.556 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.556 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd allow primary affinity = true 2023-12-04T05:39:01.556 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd allow pg remap = true 2023-12-04T05:39:01.556 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on legacy crush tunables = false 2023-12-04T05:39:01.556 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on crush straw calc version zero = false 2023-12-04T05:39:01.556 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on no sortbitwise = false 2023-12-04T05:39:01.556 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on osd down out interval zero = false 2023-12-04T05:39:01.556 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on too few osds = false 2023-12-04T05:39:01.557 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-04T05:39:01.557 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-04T05:39:01.557 INFO:teuthology.orchestra.run.smithi195.stdout: mon_allow_pool_size_one = true 2023-12-04T05:39:01.557 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.557 INFO:teuthology.orchestra.run.smithi195.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-04T05:39:01.557 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.557 INFO:teuthology.orchestra.run.smithi195.stdout: osd default data pool replay window = 5 2023-12-04T05:39:01.557 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.557 INFO:teuthology.orchestra.run.smithi195.stdout: mon allow pool delete = true 2023-12-04T05:39:01.557 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.557 INFO:teuthology.orchestra.run.smithi195.stdout: mon cluster log file level = debug 2023-12-04T05:39:01.557 INFO:teuthology.orchestra.run.smithi195.stdout: debug asserts on shutdown = true 2023-12-04T05:39:01.557 INFO:teuthology.orchestra.run.smithi195.stdout: mon health detail to clog = false 2023-12-04T05:39:01.557 INFO:teuthology.orchestra.run.smithi195.stdout: mon host = "172.21.15.107,172.21.15.195,[v2:172.21.15.195:3301,v1:172.21.15.195:6790]" 2023-12-04T05:39:01.557 INFO:teuthology.orchestra.run.smithi195.stdout: fsid = 603e856b-31ac-475a-8f35-21ab7faadd48 2023-12-04T05:39:01.557 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.558 INFO:teuthology.orchestra.run.smithi195.stdout:[osd] 2023-12-04T05:39:01.558 INFO:teuthology.orchestra.run.smithi195.stdout: osd journal size = 100 2023-12-04T05:39:01.558 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.558 INFO:teuthology.orchestra.run.smithi195.stdout: osd scrub load threshold = 5.0 2023-12-04T05:39:01.558 INFO:teuthology.orchestra.run.smithi195.stdout: osd scrub max interval = 600 2023-12-04T05:39:01.558 INFO:teuthology.orchestra.run.smithi195.stdout: osd mclock profile = high_recovery_ops 2023-12-04T05:39:01.558 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.558 INFO:teuthology.orchestra.run.smithi195.stdout: osd recover clone overlap = true 2023-12-04T05:39:01.558 INFO:teuthology.orchestra.run.smithi195.stdout: osd recovery max chunk = 1048576 2023-12-04T05:39:01.558 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.558 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug shutdown = true 2023-12-04T05:39:01.558 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug op order = true 2023-12-04T05:39:01.558 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug verify stray on activate = true 2023-12-04T05:39:01.559 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug trim objects = true 2023-12-04T05:39:01.559 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.559 INFO:teuthology.orchestra.run.smithi195.stdout: osd open classes on start = true 2023-12-04T05:39:01.559 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug pg log writeout = true 2023-12-04T05:39:01.559 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.559 INFO:teuthology.orchestra.run.smithi195.stdout: osd deep scrub update digest min age = 30 2023-12-04T05:39:01.559 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.559 INFO:teuthology.orchestra.run.smithi195.stdout: osd map max advance = 10 2023-12-04T05:39:01.559 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.559 INFO:teuthology.orchestra.run.smithi195.stdout: journal zero on create = true 2023-12-04T05:39:01.559 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.559 INFO:teuthology.orchestra.run.smithi195.stdout: filestore ondisk finisher threads = 3 2023-12-04T05:39:01.559 INFO:teuthology.orchestra.run.smithi195.stdout: filestore apply finisher threads = 3 2023-12-04T05:39:01.559 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.559 INFO:teuthology.orchestra.run.smithi195.stdout: bdev debug aio = true 2023-12-04T05:39:01.559 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug misdirected ops = true 2023-12-04T05:39:01.560 INFO:teuthology.orchestra.run.smithi195.stdout: bdev async discard = True 2023-12-04T05:39:01.560 INFO:teuthology.orchestra.run.smithi195.stdout: bdev enable discard = True 2023-12-04T05:39:01.560 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore allocator = bitmap 2023-12-04T05:39:01.560 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore block size = 96636764160 2023-12-04T05:39:01.560 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore fsck on mount = True 2023-12-04T05:39:01.560 INFO:teuthology.orchestra.run.smithi195.stdout: debug bluefs = 1/20 2023-12-04T05:39:01.560 INFO:teuthology.orchestra.run.smithi195.stdout: debug bluestore = 1/20 2023-12-04T05:39:01.560 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2023-12-04T05:39:01.560 INFO:teuthology.orchestra.run.smithi195.stdout: debug osd = 20 2023-12-04T05:39:01.560 INFO:teuthology.orchestra.run.smithi195.stdout: debug rocksdb = 4/10 2023-12-04T05:39:01.560 INFO:teuthology.orchestra.run.smithi195.stdout: filestore flush min = 0 2023-12-04T05:39:01.561 INFO:teuthology.orchestra.run.smithi195.stdout: mon client directed command retry = 5 2023-12-04T05:39:01.561 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd backfillfull_ratio = 0.85 2023-12-04T05:39:01.561 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd full ratio = 0.9 2023-12-04T05:39:01.561 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd nearfull ratio = 0.8 2023-12-04T05:39:01.561 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject delay max = 1 2023-12-04T05:39:01.561 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject delay probability = 0.005 2023-12-04T05:39:01.561 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject delay type = client mds 2023-12-04T05:39:01.561 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject socket failures = 2500 2023-12-04T05:39:01.561 INFO:teuthology.orchestra.run.smithi195.stdout: osd blocked scrub grace period = 3600 2023-12-04T05:39:01.561 INFO:teuthology.orchestra.run.smithi195.stdout: osd failsafe full ratio = 0.95 2023-12-04T05:39:01.561 INFO:teuthology.orchestra.run.smithi195.stdout: osd max markdown count = 1000 2023-12-04T05:39:01.561 INFO:teuthology.orchestra.run.smithi195.stdout: osd objectstore = bluestore 2023-12-04T05:39:01.562 INFO:teuthology.orchestra.run.smithi195.stdout: osd op complaint time = 180 2023-12-04T05:39:01.562 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.562 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr] 2023-12-04T05:39:01.562 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2023-12-04T05:39:01.562 INFO:teuthology.orchestra.run.smithi195.stdout: debug mgr = 20 2023-12-04T05:39:01.562 INFO:teuthology.orchestra.run.smithi195.stdout: debug mon = 20 2023-12-04T05:39:01.562 INFO:teuthology.orchestra.run.smithi195.stdout: debug auth = 20 2023-12-04T05:39:01.562 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min pgs per osd = 4 2023-12-04T05:39:01.562 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min bytes per osd = 10 2023-12-04T05:39:01.562 INFO:teuthology.orchestra.run.smithi195.stdout: mgr/telemetry/nag = false 2023-12-04T05:39:01.562 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.562 INFO:teuthology.orchestra.run.smithi195.stdout:[mon] 2023-12-04T05:39:01.562 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2023-12-04T05:39:01.562 INFO:teuthology.orchestra.run.smithi195.stdout: debug mon = 20 2023-12-04T05:39:01.562 INFO:teuthology.orchestra.run.smithi195.stdout: debug paxos = 20 2023-12-04T05:39:01.563 INFO:teuthology.orchestra.run.smithi195.stdout: debug auth = 20 2023-12-04T05:39:01.563 INFO:teuthology.orchestra.run.smithi195.stdout: mon data avail warn = 5 2023-12-04T05:39:01.563 INFO:teuthology.orchestra.run.smithi195.stdout: mon mgr mkfs grace = 240 2023-12-04T05:39:01.563 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min pgs per osd = 4 2023-12-04T05:39:01.563 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd reporter subtree level = osd 2023-12-04T05:39:01.563 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd prime pg temp = true 2023-12-04T05:39:01.563 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min bytes per osd = 10 2023-12-04T05:39:01.563 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.563 INFO:teuthology.orchestra.run.smithi195.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-04T05:39:01.563 INFO:teuthology.orchestra.run.smithi195.stdout: auth mon ticket ttl = 660 # 11m 2023-12-04T05:39:01.564 INFO:teuthology.orchestra.run.smithi195.stdout: auth service ticket ttl = 240 # 4m 2023-12-04T05:39:01.564 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.564 INFO:teuthology.orchestra.run.smithi195.stdout: # don't complain about insecure global_id in the test suite 2023-12-04T05:39:01.564 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-04T05:39:01.564 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-04T05:39:01.564 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.564 INFO:teuthology.orchestra.run.smithi195.stdout: # 1m isn't quite enough 2023-12-04T05:39:01.564 INFO:teuthology.orchestra.run.smithi195.stdout: mon_down_mkfs_grace = 2m 2023-12-04T05:39:01.564 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.564 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_filestore_osds = false 2023-12-04T05:39:01.564 INFO:teuthology.orchestra.run.smithi195.stdout: mon op complaint time = 120 2023-12-04T05:39:01.564 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-12-04T05:39:01.564 INFO:teuthology.orchestra.run.smithi195.stdout:[client] 2023-12-04T05:39:01.565 INFO:teuthology.orchestra.run.smithi195.stdout: rgw cache enabled = true 2023-12-04T05:39:01.565 INFO:teuthology.orchestra.run.smithi195.stdout: rgw enable ops log = true 2023-12-04T05:39:01.565 INFO:teuthology.orchestra.run.smithi195.stdout: rgw enable usage log = true 2023-12-04T05:39:01.565 INFO:teuthology.orchestra.run.smithi195.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-04T05:39:01.565 INFO:teuthology.orchestra.run.smithi195.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-04T05:39:01.565 INFO:teuthology.orchestra.run.smithi195.stdout: client mount timeout = 600 2023-12-04T05:39:01.565 INFO:teuthology.orchestra.run.smithi195.stdout: client_shutdown_timeout = 120 2023-12-04T05:39:01.565 INFO:teuthology.orchestra.run.smithi195.stdout: debug client = 20 2023-12-04T05:39:01.565 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2023-12-04T05:39:01.566 INFO:teuthology.orchestra.run.smithi195.stdout: rados mon op timeout = 900 2023-12-04T05:39:01.566 INFO:teuthology.orchestra.run.smithi195.stdout: rados osd op timeout = 900 2023-12-04T05:39:01.566 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.a] 2023-12-04T05:39:01.566 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.b] 2023-12-04T05:39:01.566 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.c] 2023-12-04T05:39:01.566 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.a] 2023-12-04T05:39:01.566 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.c] 2023-12-04T05:39:01.567 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.e] 2023-12-04T05:39:01.567 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.b] 2023-12-04T05:39:01.567 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.d] 2023-12-04T05:39:01.567 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.f] 2023-12-04T05:39:01.567 INFO:teuthology.orchestra.run.smithi195.stdout:[mds] 2023-12-04T05:39:01.567 INFO:teuthology.orchestra.run.smithi195.stdout: debug mds = 20 2023-12-04T05:39:01.567 INFO:teuthology.orchestra.run.smithi195.stdout: debug mds balancer = 20 2023-12-04T05:39:01.567 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2023-12-04T05:39:01.568 INFO:teuthology.orchestra.run.smithi195.stdout: mds bal fragment size max = 10000 2023-12-04T05:39:01.568 INFO:teuthology.orchestra.run.smithi195.stdout: mds bal merge size = 5 2023-12-04T05:39:01.568 INFO:teuthology.orchestra.run.smithi195.stdout: mds bal split bits = 3 2023-12-04T05:39:01.568 INFO:teuthology.orchestra.run.smithi195.stdout: mds bal split size = 100 2023-12-04T05:39:01.568 INFO:teuthology.orchestra.run.smithi195.stdout: mds debug frag = True 2023-12-04T05:39:01.568 INFO:teuthology.orchestra.run.smithi195.stdout: mds debug scatterstat = True 2023-12-04T05:39:01.568 INFO:teuthology.orchestra.run.smithi195.stdout: mds oft prefetch dirfrags = True 2023-12-04T05:39:01.569 INFO:teuthology.orchestra.run.smithi195.stdout: mds op complaint time = 180 2023-12-04T05:39:01.569 INFO:teuthology.orchestra.run.smithi195.stdout: mds verify scatter = True 2023-12-04T05:39:01.569 INFO:teuthology.orchestra.run.smithi195.stdout: mds_dir_prefetch = False 2023-12-04T05:39:01.569 INFO:teuthology.orchestra.run.smithi195.stdout: mds_sleep_rank_change = 5000000.0 2023-12-04T05:39:01.569 INFO:teuthology.orchestra.run.smithi195.stdout: mon client directed command retry = 5 2023-12-04T05:39:01.569 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject delay max = 1 2023-12-04T05:39:01.569 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject delay probability = 0.005 2023-12-04T05:39:01.570 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject delay type = client mds osd 2023-12-04T05:39:01.570 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject socket failures = 2500 2023-12-04T05:39:01.570 INFO:teuthology.orchestra.run.smithi195.stdout: osd op complaint time = 180 2023-12-04T05:39:01.570 INFO:teuthology.orchestra.run.smithi195.stdout: rados mon op timeout = 900 2023-12-04T05:39:01.570 INFO:teuthology.orchestra.run.smithi195.stdout: rados osd op timeout = 900 2023-12-04T05:39:01.570 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-04T05:39:01.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-04T05:39:01.647 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-04T05:39:01.647 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:39:01.647 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-04T05:39:01.652 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:39:01.653 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-04T05:39:01.703 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi107.front.sepia.ceph.com 2023-12-04T05:39:01.703 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:39:01.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-04T05:39:01.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-04T05:39:01.768 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:39:01.768 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-04T05:39:01.815 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi195.front.sepia.ceph.com 2023-12-04T05:39:01.816 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:39:01.816 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-04T05:39:01.816 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-04T05:39:01.838 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:39:01.839 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-04T05:39:01.887 INFO:tasks.ceph:Setting up mon nodes... 2023-12-04T05:39:01.887 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-04T05:39:01.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-12-04T05:39:01.931 INFO:teuthology.orchestra.run.smithi107.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-12-04T05:39:01.934 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-04T05:39:01.979 INFO:teuthology.orchestra.run.smithi195.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-04T05:39:01.987 INFO:tasks.ceph:Setting up mds nodes... 2023-12-04T05:39:01.987 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-12-04T05:39:02.028 INFO:teuthology.orchestra.run.smithi107.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-12-04T05:39:02.031 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-12-04T05:39:02.085 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-12-04T05:39:02.166 INFO:teuthology.orchestra.run.smithi107.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-12-04T05:39:02.169 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-12-04T05:39:02.222 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2023-12-04T05:39:02.306 INFO:teuthology.orchestra.run.smithi107.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2023-12-04T05:39:02.309 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2023-12-04T05:39:02.363 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-12-04T05:39:02.408 INFO:teuthology.orchestra.run.smithi195.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-12-04T05:39:02.411 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-12-04T05:39:02.466 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-12-04T05:39:02.551 INFO:teuthology.orchestra.run.smithi195.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-12-04T05:39:02.554 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-12-04T05:39:02.610 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2023-12-04T05:39:02.695 INFO:teuthology.orchestra.run.smithi195.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2023-12-04T05:39:02.697 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2023-12-04T05:39:02.755 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-04T05:39:02.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-04T05:39:02.792 INFO:teuthology.orchestra.run.smithi107.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-04T05:39:02.802 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-04T05:39:02.803 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi195.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-04T05:39:02.803 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-04T05:39:02.858 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-04T05:39:02.859 INFO:tasks.ceph:role: osd.0 2023-12-04T05:39:02.859 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi107.front.sepia.ceph.com 2023-12-04T05:39:02.859 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-04T05:39:02.924 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T05:39:02.924 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T05:39:02.924 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T05:39:02.925 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-04T05:39:02.925 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T05:39:02.925 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2023-12-04T05:39:02.925 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T05:39:02.925 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-04T05:39:02.925 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T05:39:02.925 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T05:39:03.382 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2023-12-04T05:39:03.385 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2023-12-04T05:39:03.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-04T05:39:03.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-04T05:39:03.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T05:39:03.496 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: /sbin/restorecon: command not found 2023-12-04T05:39:03.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-04T05:39:03.549 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-04T05:39:03.549 INFO:tasks.ceph:role: osd.1 2023-12-04T05:39:03.549 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi107.front.sepia.ceph.com 2023-12-04T05:39:03.549 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-04T05:39:03.614 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T05:39:03.614 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T05:39:03.614 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T05:39:03.614 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-04T05:39:03.614 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T05:39:03.614 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2023-12-04T05:39:03.614 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T05:39:03.614 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-04T05:39:03.614 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T05:39:03.614 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T05:39:04.078 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2023-12-04T05:39:04.080 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2023-12-04T05:39:04.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-04T05:39:04.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-04T05:39:04.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T05:39:04.204 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: /sbin/restorecon: command not found 2023-12-04T05:39:04.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-04T05:39:04.257 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-04T05:39:04.257 INFO:tasks.ceph:role: osd.2 2023-12-04T05:39:04.257 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi107.front.sepia.ceph.com 2023-12-04T05:39:04.257 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-04T05:39:04.319 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T05:39:04.319 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T05:39:04.319 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T05:39:04.319 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-04T05:39:04.319 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T05:39:04.319 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2023-12-04T05:39:04.319 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T05:39:04.319 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-04T05:39:04.319 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T05:39:04.320 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T05:39:04.802 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2023-12-04T05:39:04.804 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2023-12-04T05:39:04.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-04T05:39:04.948 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-04T05:39:04.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T05:39:04.959 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: /sbin/restorecon: command not found 2023-12-04T05:39:04.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-12-04T05:39:05.015 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-04T05:39:05.015 INFO:tasks.ceph:role: osd.3 2023-12-04T05:39:05.016 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi107.front.sepia.ceph.com 2023-12-04T05:39:05.016 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-04T05:39:05.079 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T05:39:05.079 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T05:39:05.079 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T05:39:05.079 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-04T05:39:05.079 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T05:39:05.079 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2023-12-04T05:39:05.080 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T05:39:05.080 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-04T05:39:05.080 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T05:39:05.080 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T05:39:05.514 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2023-12-04T05:39:05.516 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2023-12-04T05:39:05.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-12-04T05:39:05.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-12-04T05:39:05.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T05:39:05.657 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: /sbin/restorecon: command not found 2023-12-04T05:39:05.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-04T05:39:05.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:05.765+0000 7fea77895640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-12-04T05:39:05.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:05.769+0000 7fea77895640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-12-04T05:39:05.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:05.769+0000 7fea77895640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-04T05:39:05.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:05.769+0000 7fea77895640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-04T05:39:05.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:05.769+0000 7fea77895640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-04T05:39:05.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:05.769+0000 7fea77895640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-12-04T05:39:08.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-04T05:39:08.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-04T05:39:08.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:08.881+0000 7f9698d7b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-12-04T05:39:08.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:08.885+0000 7f9698d7b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-12-04T05:39:08.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:08.885+0000 7f9698d7b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-04T05:39:08.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:08.885+0000 7f9698d7b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-04T05:39:08.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:08.885+0000 7f9698d7b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-04T05:39:08.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:08.885+0000 7f9698d7b640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-12-04T05:39:11.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-04T05:39:11.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-04T05:39:12.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:12.053+0000 7fdd1bce4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-12-04T05:39:12.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:12.053+0000 7fdd1bce4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-12-04T05:39:12.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:12.053+0000 7fdd1bce4640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-04T05:39:12.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:12.053+0000 7fdd1bce4640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-04T05:39:12.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:12.053+0000 7fdd1bce4640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-04T05:39:12.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:12.053+0000 7fdd1bce4640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-12-04T05:39:15.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-04T05:39:15.111 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-04T05:39:15.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:15.193+0000 7f533bd25640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-12-04T05:39:15.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:15.193+0000 7f533bd25640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-12-04T05:39:15.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:15.193+0000 7f533bd25640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-04T05:39:15.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:15.193+0000 7f533bd25640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-04T05:39:15.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:15.193+0000 7f533bd25640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-04T05:39:15.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:15.193+0000 7f533bd25640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-12-04T05:39:18.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-12-04T05:39:18.268 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-12-04T05:39:18.301 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-04T05:39:18.301 INFO:tasks.ceph:role: osd.4 2023-12-04T05:39:18.302 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi195.front.sepia.ceph.com 2023-12-04T05:39:18.302 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-04T05:39:18.358 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T05:39:18.358 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T05:39:18.358 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T05:39:18.358 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-04T05:39:18.358 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T05:39:18.358 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2023-12-04T05:39:18.359 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T05:39:18.359 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-04T05:39:18.359 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T05:39:18.359 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T05:39:18.386 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2023-12-04T05:39:18.387 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2023-12-04T05:39:18.387 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-12-04T05:39:18.445 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-12-04T05:39:18.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T05:39:18.497 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: /sbin/restorecon: command not found 2023-12-04T05:39:18.498 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-12-04T05:39:18.555 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-04T05:39:18.555 INFO:tasks.ceph:role: osd.5 2023-12-04T05:39:18.555 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi195.front.sepia.ceph.com 2023-12-04T05:39:18.555 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-04T05:39:18.610 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T05:39:18.610 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T05:39:18.610 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T05:39:18.610 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-04T05:39:18.610 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T05:39:18.611 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2023-12-04T05:39:18.611 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T05:39:18.611 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-04T05:39:18.611 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T05:39:18.611 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T05:39:18.636 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2023-12-04T05:39:18.638 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2023-12-04T05:39:18.638 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-12-04T05:39:18.699 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-12-04T05:39:18.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T05:39:18.751 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: /sbin/restorecon: command not found 2023-12-04T05:39:18.751 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-12-04T05:39:18.804 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-04T05:39:18.804 INFO:tasks.ceph:role: osd.6 2023-12-04T05:39:18.804 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi195.front.sepia.ceph.com 2023-12-04T05:39:18.805 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-04T05:39:18.867 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T05:39:18.868 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T05:39:18.868 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T05:39:18.868 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-04T05:39:18.868 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T05:39:18.868 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2023-12-04T05:39:18.868 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T05:39:18.868 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-04T05:39:18.868 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T05:39:18.868 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T05:39:18.892 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2023-12-04T05:39:18.894 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2023-12-04T05:39:18.894 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-12-04T05:39:18.955 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-12-04T05:39:19.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T05:39:19.007 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: /sbin/restorecon: command not found 2023-12-04T05:39:19.008 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-12-04T05:39:19.066 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-04T05:39:19.066 INFO:tasks.ceph:role: osd.7 2023-12-04T05:39:19.067 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi195.front.sepia.ceph.com 2023-12-04T05:39:19.067 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-04T05:39:19.121 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T05:39:19.121 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T05:39:19.121 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T05:39:19.122 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-04T05:39:19.122 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T05:39:19.122 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2023-12-04T05:39:19.122 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T05:39:19.122 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-04T05:39:19.122 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T05:39:19.122 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T05:39:19.147 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2023-12-04T05:39:19.149 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2023-12-04T05:39:19.150 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-12-04T05:39:19.212 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-12-04T05:39:19.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T05:39:19.264 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: /sbin/restorecon: command not found 2023-12-04T05:39:19.265 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-04T05:39:19.378 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:19.373+0000 7f12cc784640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-12-04T05:39:19.378 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:19.377+0000 7f12cc784640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-12-04T05:39:19.378 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:19.377+0000 7f12cc784640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-04T05:39:19.379 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:19.377+0000 7f12cc784640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-04T05:39:19.379 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:19.377+0000 7f12cc784640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-04T05:39:19.379 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:19.377+0000 7f12cc784640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-12-04T05:39:22.379 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-12-04T05:39:22.393 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-04T05:39:22.473 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:22.469+0000 7f7ce2621640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-12-04T05:39:22.474 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:22.469+0000 7f7ce2621640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-12-04T05:39:22.474 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:22.469+0000 7f7ce2621640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-04T05:39:22.474 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:22.469+0000 7f7ce2621640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-04T05:39:22.474 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:22.469+0000 7f7ce2621640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-04T05:39:22.474 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:22.473+0000 7f7ce2621640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-12-04T05:39:25.478 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-12-04T05:39:25.539 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-04T05:39:25.623 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:25.621+0000 7fa88eec3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-12-04T05:39:25.623 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:25.621+0000 7fa88eec3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-12-04T05:39:25.623 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:25.621+0000 7fa88eec3640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-04T05:39:25.623 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:25.621+0000 7fa88eec3640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-04T05:39:25.623 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:25.621+0000 7fa88eec3640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-04T05:39:25.623 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:25.621+0000 7fa88eec3640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-12-04T05:39:28.618 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-12-04T05:39:28.675 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-04T05:39:28.757 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:28.753+0000 7fc0c8f10640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-12-04T05:39:28.757 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:28.753+0000 7fc0c8f10640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-12-04T05:39:28.758 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:28.753+0000 7fc0c8f10640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-04T05:39:28.758 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:28.753+0000 7fc0c8f10640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-04T05:39:28.758 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:28.753+0000 7fc0c8f10640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-04T05:39:28.758 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:28.757+0000 7fc0c8f10640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-12-04T05:39:29.749 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:39:29.753 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:39:31.773 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-12-04T05:39:31.787 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-04T05:39:31.787 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:39:31.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-12-04T05:39:31.802 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:39:31.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-12-04T05:39:31.855 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:39:31.855 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-12-04T05:39:31.910 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:39:31.910 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2023-12-04T05:39:31.966 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:39:31.966 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-04T05:39:32.022 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:39:32.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-04T05:39:32.078 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:39:32.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-04T05:39:32.135 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:39:32.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-12-04T05:39:32.191 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:39:32.191 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-04T05:39:32.205 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:39:32.205 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-12-04T05:39:32.261 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:39:32.261 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-12-04T05:39:32.317 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:39:32.317 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2023-12-04T05:39:32.373 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:39:32.373 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-12-04T05:39:32.429 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:39:32.429 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-12-04T05:39:32.481 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:39:32.481 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-12-04T05:39:32.536 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:39:32.537 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-12-04T05:39:32.589 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:39:32.589 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-04T05:39:32.594 INFO:tasks.ceph:Adding keys to all mons... 2023-12-04T05:39:32.594 DEBUG:teuthology.orchestra.run.smithi107:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-04T05:39:32.640 DEBUG:teuthology.orchestra.run.smithi195:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-04T05:39:32.650 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr.y] 2023-12-04T05:39:32.650 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD1ZW1lD36QNxAAcqM92HKAoS2xBEQz8ZHafA== 2023-12-04T05:39:32.650 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.a] 2023-12-04T05:39:32.650 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD2ZW1ldADKARAAizI/5eLZ74vmMCmwJpt4Fg== 2023-12-04T05:39:32.650 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.c] 2023-12-04T05:39:32.650 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD2ZW1lKvr9CRAAvj3W2JELHR1NWEUfsJc8jw== 2023-12-04T05:39:32.650 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.e] 2023-12-04T05:39:32.650 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD2ZW1lbs1cEhAAlsPU9ypReapKjYxD7Pnkmg== 2023-12-04T05:39:32.650 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.0] 2023-12-04T05:39:32.651 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD5ZW1lBdn/LRAARmWONRqWjPGimHZft9V3nw== 2023-12-04T05:39:32.651 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.1] 2023-12-04T05:39:32.651 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD8ZW1l2HzrNBAAT+iyK4vgQaErARFLipDnvw== 2023-12-04T05:39:32.651 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.2] 2023-12-04T05:39:32.651 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQAAZm1lg7ZoAxAAXVuYXQ44vnQf++3mQNbhCw== 2023-12-04T05:39:32.651 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.3] 2023-12-04T05:39:32.651 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQADZm1lj0HBCxAAQdX8mN57HgjGAzras9Jnew== 2023-12-04T05:39:32.651 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr.x] 2023-12-04T05:39:32.651 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD1ZW1lhvtmOhAAeX5rEfW63TyVoXf5y1i6rw== 2023-12-04T05:39:32.651 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.b] 2023-12-04T05:39:32.651 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD2ZW1lskRmGBAAGg+K/9h8XnJvz7oD69TP5Q== 2023-12-04T05:39:32.651 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.d] 2023-12-04T05:39:32.651 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD2ZW1lr4jxIBAAMGhcW50Garbv93/k2mQ8OQ== 2023-12-04T05:39:32.651 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.f] 2023-12-04T05:39:32.651 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD2ZW1lbzh7KRAAy18FkWEbPjwPHFKtaWRblQ== 2023-12-04T05:39:32.652 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.4] 2023-12-04T05:39:32.652 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQAHZm1lRwuUFhAA7M/Goy+HDV/nuKjdUICQ8g== 2023-12-04T05:39:32.652 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.5] 2023-12-04T05:39:32.652 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQAKZm1ldKlGHBAAzH1QSsWhX2oD44fYuvCtiQ== 2023-12-04T05:39:32.652 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.6] 2023-12-04T05:39:32.652 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQANZm1lf7grJRAAUeClqKq0ORaZ8HgQXcceHw== 2023-12-04T05:39:32.652 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.7] 2023-12-04T05:39:32.652 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQAQZm1lKrsyLRAAKEEUOyLnwO8HH3TAxwR7dQ== 2023-12-04T05:39:32.652 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2023-12-04T05:39:32.652 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD2ZW1l2s9ULxAAcxsyVtpiM62iYrY+bVN3iQ== 2023-12-04T05:39:32.653 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr.y] 2023-12-04T05:39:32.654 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQD1ZW1lD36QNxAAcqM92HKAoS2xBEQz8ZHafA== 2023-12-04T05:39:32.654 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.a] 2023-12-04T05:39:32.654 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQD2ZW1ldADKARAAizI/5eLZ74vmMCmwJpt4Fg== 2023-12-04T05:39:32.654 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.c] 2023-12-04T05:39:32.654 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQD2ZW1lKvr9CRAAvj3W2JELHR1NWEUfsJc8jw== 2023-12-04T05:39:32.654 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.e] 2023-12-04T05:39:32.654 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQD2ZW1lbs1cEhAAlsPU9ypReapKjYxD7Pnkmg== 2023-12-04T05:39:32.654 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.0] 2023-12-04T05:39:32.654 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQD5ZW1lBdn/LRAARmWONRqWjPGimHZft9V3nw== 2023-12-04T05:39:32.654 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.1] 2023-12-04T05:39:32.654 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQD8ZW1l2HzrNBAAT+iyK4vgQaErARFLipDnvw== 2023-12-04T05:39:32.654 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.2] 2023-12-04T05:39:32.655 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAAZm1lg7ZoAxAAXVuYXQ44vnQf++3mQNbhCw== 2023-12-04T05:39:32.655 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.3] 2023-12-04T05:39:32.655 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQADZm1lj0HBCxAAQdX8mN57HgjGAzras9Jnew== 2023-12-04T05:39:32.655 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr.x] 2023-12-04T05:39:32.655 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQD1ZW1lhvtmOhAAeX5rEfW63TyVoXf5y1i6rw== 2023-12-04T05:39:32.655 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.b] 2023-12-04T05:39:32.655 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQD2ZW1lskRmGBAAGg+K/9h8XnJvz7oD69TP5Q== 2023-12-04T05:39:32.655 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.d] 2023-12-04T05:39:32.655 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQD2ZW1lr4jxIBAAMGhcW50Garbv93/k2mQ8OQ== 2023-12-04T05:39:32.655 INFO:teuthology.orchestra.run.smithi195.stdout:[mds.f] 2023-12-04T05:39:32.655 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQD2ZW1lbzh7KRAAy18FkWEbPjwPHFKtaWRblQ== 2023-12-04T05:39:32.655 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.4] 2023-12-04T05:39:32.655 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAHZm1lRwuUFhAA7M/Goy+HDV/nuKjdUICQ8g== 2023-12-04T05:39:32.656 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.5] 2023-12-04T05:39:32.656 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAKZm1ldKlGHBAAzH1QSsWhX2oD44fYuvCtiQ== 2023-12-04T05:39:32.656 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.6] 2023-12-04T05:39:32.656 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQANZm1lf7grJRAAUeClqKq0ORaZ8HgQXcceHw== 2023-12-04T05:39:32.656 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.7] 2023-12-04T05:39:32.656 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAQZm1lKrsyLRAAKEEUOyLnwO8HH3TAxwR7dQ== 2023-12-04T05:39:32.656 INFO:teuthology.orchestra.run.smithi195.stdout:[client.0] 2023-12-04T05:39:32.656 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQD2ZW1l2s9ULxAAcxsyVtpiM62iYrY+bVN3iQ== 2023-12-04T05:39:32.656 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-04T05:39:32.696 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-04T05:39:32.741 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T05:39:32.784 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T05:39:32.832 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T05:39:32.872 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T05:39:32.921 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T05:39:32.964 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T05:39:33.009 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T05:39:33.012 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T05:39:33.088 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T05:39:33.096 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T05:39:33.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T05:39:33.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T05:39:33.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T05:39:33.264 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T05:39:33.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-04T05:39:33.352 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-04T05:39:33.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T05:39:33.432 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T05:39:33.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T05:39:33.518 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T05:39:33.601 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T05:39:33.608 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T05:39:33.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T05:39:33.696 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T05:39:33.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T05:39:33.776 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T05:39:33.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T05:39:33.860 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T05:39:33.929 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T05:39:33.945 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T05:39:34.009 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T05:39:34.012 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T05:39:34.099 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-04T05:39:34.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-04T05:39:34.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-04T05:39:34.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-04T05:39:34.220 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-12-04T05:39:34.234 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-04T05:39:34.332 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-12-04T05:39:34.343 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-12-04T05:39:34.396 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-04T05:39:34.486 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-12-04T05:39:34.496 DEBUG:teuthology.orchestra.run.smithi107:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-04T05:39:34.499 DEBUG:teuthology.orchestra.run.smithi195:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-04T05:39:34.541 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-04T05:39:34.542 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-04T05:39:34.542 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-12-04T05:39:34.548 INFO:tasks.ceph.mon.a:Started 2023-12-04T05:39:34.548 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-04T05:39:34.548 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-12-04T05:39:34.587 INFO:tasks.ceph.mon.b:Started 2023-12-04T05:39:34.587 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-04T05:39:34.587 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-12-04T05:39:34.589 INFO:tasks.ceph.mon.c:Started 2023-12-04T05:39:34.589 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-04T05:39:34.590 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-12-04T05:39:34.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-12-04T05:39:34.592 INFO:tasks.ceph.mgr.y:Started 2023-12-04T05:39:34.592 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-04T05:39:34.593 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-12-04T05:39:34.595 INFO:tasks.ceph.mgr.x:Started 2023-12-04T05:39:34.595 INFO:tasks.ceph:Setting crush tunables to default 2023-12-04T05:39:34.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd crush tunables default 2023-12-04T05:39:34.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.705+0000 7fe190d4d640 1 Processor -- start 2023-12-04T05:39:34.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.705+0000 7fe190d4d640 1 -- start start 2023-12-04T05:39:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe190d4d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe18c151fb0 0x7fe18c0724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe190d4d640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe18c0c9a10 0x7fe18c0ca450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe190d4d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe18c0ca990 0x7fe18c1581c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe190d4d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe18c157da0 con 0x7fe18c0ca990 2023-12-04T05:39:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe190d4d640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe18c0c65c0 con 0x7fe18c0c9a10 2023-12-04T05:39:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe190d4d640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c0c6ba0 con 0x7fe18c151fb0 2023-12-04T05:39:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe18ad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe18c0ca990 0x7fe18c1581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe18ad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe18c0ca990 0x7fe18c1581c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36496/0 (socket says 172.21.15.107:36496) 2023-12-04T05:39:34.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe18ad9d640 1 -- 172.21.15.107:0/4204014815 learned_addr learned my addr 172.21.15.107:0/4204014815 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:39:34.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe189d9b640 1 --2- 172.21.15.107:0/4204014815 >> v2:172.21.15.195:3300/0 conn(0x7fe18c0c9a10 0x7fe18c0ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:34.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe18a59c640 1 --2- 172.21.15.107:0/4204014815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe18c151fb0 0x7fe18c0724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:34.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe18ad9d640 1 -- 172.21.15.107:0/4204014815 >> v2:172.21.15.107:3300/0 conn(0x7fe18c0ca990 msgr2=0x7fe18c1581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-04T05:39:34.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe18ad9d640 1 -- 172.21.15.107:0/4204014815 >> v2:172.21.15.107:3300/0 conn(0x7fe18c0ca990 msgr2=0x7fe18c1581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T05:39:34.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe18ad9d640 1 --2- 172.21.15.107:0/4204014815 >> v2:172.21.15.107:3300/0 conn(0x7fe18c0ca990 0x7fe18c1581c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T05:39:34.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe18ad9d640 1 --2- 172.21.15.107:0/4204014815 >> v2:172.21.15.107:3300/0 conn(0x7fe18c0ca990 0x7fe18c1581c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:39:34.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe189d9b640 1 -- 172.21.15.107:0/4204014815 >> v2:172.21.15.195:3300/0 conn(0x7fe18c0c9a10 msgr2=0x7fe18c0ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-04T05:39:34.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe189d9b640 1 -- 172.21.15.107:0/4204014815 >> v2:172.21.15.195:3300/0 conn(0x7fe18c0c9a10 msgr2=0x7fe18c0ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T05:39:34.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe189d9b640 1 --2- 172.21.15.107:0/4204014815 >> v2:172.21.15.195:3300/0 conn(0x7fe18c0c9a10 0x7fe18c0ca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T05:39:34.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe189d9b640 1 --2- 172.21.15.107:0/4204014815 >> v2:172.21.15.195:3300/0 conn(0x7fe18c0c9a10 0x7fe18c0ca450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:39:34.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe18a59c640 1 -- 172.21.15.107:0/4204014815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe18c151fb0 msgr2=0x7fe18c0724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-04T05:39:34.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe18a59c640 1 -- 172.21.15.107:0/4204014815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe18c151fb0 msgr2=0x7fe18c0724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T05:39:34.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe18a59c640 1 --2- 172.21.15.107:0/4204014815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe18c151fb0 0x7fe18c0724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T05:39:34.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.709+0000 7fe18a59c640 1 --2- 172.21.15.107:0/4204014815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe18c151fb0 0x7fe18c0724c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:39:34.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.913+0000 7fe18ad9d640 1 --2- 172.21.15.107:0/4204014815 >> v2:172.21.15.107:3300/0 conn(0x7fe18c0ca990 0x7fe18c1581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:34.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.913+0000 7fe18a59c640 1 --2- 172.21.15.107:0/4204014815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe18c151fb0 0x7fe18c0724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:34.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.913+0000 7fe189d9b640 1 --2- 172.21.15.107:0/4204014815 >> v2:172.21.15.195:3300/0 conn(0x7fe18c0c9a10 0x7fe18c0ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:34.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.913+0000 7fe18ad9d640 1 -- 172.21.15.107:0/4204014815 >> v2:172.21.15.107:3300/0 conn(0x7fe18c0ca990 msgr2=0x7fe18c1581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-04T05:39:34.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.913+0000 7fe18ad9d640 1 -- 172.21.15.107:0/4204014815 >> v2:172.21.15.107:3300/0 conn(0x7fe18c0ca990 msgr2=0x7fe18c1581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T05:39:34.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.913+0000 7fe18ad9d640 1 --2- 172.21.15.107:0/4204014815 >> v2:172.21.15.107:3300/0 conn(0x7fe18c0ca990 0x7fe18c1581c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T05:39:34.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.913+0000 7fe18ad9d640 1 --2- 172.21.15.107:0/4204014815 >> v2:172.21.15.107:3300/0 conn(0x7fe18c0ca990 0x7fe18c1581c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:39:34.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.913+0000 7fe18a59c640 1 -- 172.21.15.107:0/4204014815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe18c151fb0 msgr2=0x7fe18c0724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-04T05:39:34.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.913+0000 7fe18a59c640 1 -- 172.21.15.107:0/4204014815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe18c151fb0 msgr2=0x7fe18c0724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T05:39:34.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.913+0000 7fe18a59c640 1 --2- 172.21.15.107:0/4204014815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe18c151fb0 0x7fe18c0724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T05:39:34.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.913+0000 7fe189d9b640 1 -- 172.21.15.107:0/4204014815 >> v2:172.21.15.195:3300/0 conn(0x7fe18c0c9a10 msgr2=0x7fe18c0ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-04T05:39:34.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.913+0000 7fe189d9b640 1 -- 172.21.15.107:0/4204014815 >> v2:172.21.15.195:3300/0 conn(0x7fe18c0c9a10 msgr2=0x7fe18c0ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T05:39:34.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.913+0000 7fe189d9b640 1 --2- 172.21.15.107:0/4204014815 >> v2:172.21.15.195:3300/0 conn(0x7fe18c0c9a10 0x7fe18c0ca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T05:39:34.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.913+0000 7fe18a59c640 1 --2- 172.21.15.107:0/4204014815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe18c151fb0 0x7fe18c0724c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:39:34.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:34.913+0000 7fe189d9b640 1 --2- 172.21.15.107:0/4204014815 >> v2:172.21.15.195:3300/0 conn(0x7fe18c0c9a10 0x7fe18c0ca450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:39:35.151 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:35.149+0000 7f7cb9017180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-04T05:39:35.165 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:35.161+0000 7fdb3fdd0180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-04T05:39:35.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.317+0000 7fe18ad9d640 1 --2- 172.21.15.107:0/4204014815 >> v2:172.21.15.107:3300/0 conn(0x7fe18c0ca990 0x7fe18c1581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:35.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.317+0000 7fe18a59c640 1 --2- 172.21.15.107:0/4204014815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe18c151fb0 0x7fe18c0724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:35.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.317+0000 7fe189d9b640 1 --2- 172.21.15.107:0/4204014815 >> v2:172.21.15.195:3300/0 conn(0x7fe18c0c9a10 0x7fe18c0ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:35.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.317+0000 7fe18ad9d640 1 -- 172.21.15.107:0/4204014815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe18c151fb0 msgr2=0x7fe18c0724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:35.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.317+0000 7fe18ad9d640 1 --2- 172.21.15.107:0/4204014815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe18c151fb0 0x7fe18c0724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:35.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.317+0000 7fe18ad9d640 1 -- 172.21.15.107:0/4204014815 >> v2:172.21.15.195:3300/0 conn(0x7fe18c0c9a10 msgr2=0x7fe18c0ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:35.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.317+0000 7fe18ad9d640 1 --2- 172.21.15.107:0/4204014815 >> v2:172.21.15.195:3300/0 conn(0x7fe18c0c9a10 0x7fe18c0ca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:35.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.317+0000 7fe18ad9d640 1 -- 172.21.15.107:0/4204014815 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18c072e80 con 0x7fe18c0ca990 2023-12-04T05:39:35.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.321+0000 7fe18ad9d640 1 --2- 172.21.15.107:0/4204014815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe18c0ca990 0x7fe18c1581c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0ce6b0 tx=0x7fe17c01caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6662cbc626fda37 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:35.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.321+0000 7fe18959a640 1 -- 172.21.15.107:0/4204014815 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17c0a5aa0 con 0x7fe18c0ca990 2023-12-04T05:39:35.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.321+0000 7fe18959a640 1 -- 172.21.15.107:0/4204014815 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe17c0a9070 con 0x7fe18c0ca990 2023-12-04T05:39:35.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.321+0000 7fe18959a640 1 -- 172.21.15.107:0/4204014815 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17c0a5ea0 con 0x7fe18c0ca990 2023-12-04T05:39:35.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.325+0000 7fe190d4d640 1 -- 172.21.15.107:0/4204014815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe18c0ca990 msgr2=0x7fe18c1581c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:35.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.325+0000 7fe190d4d640 1 --2- 172.21.15.107:0/4204014815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe18c0ca990 0x7fe18c1581c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0ce6b0 tx=0x7fe17c01caf0 comp rx=0 tx=0).stop 2023-12-04T05:39:35.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.325+0000 7fe190d4d640 1 -- 172.21.15.107:0/4204014815 shutdown_connections 2023-12-04T05:39:35.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.325+0000 7fe190d4d640 1 --2- 172.21.15.107:0/4204014815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe18c0ca990 0x7fe18c1581c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:35.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.325+0000 7fe190d4d640 1 --2- 172.21.15.107:0/4204014815 >> v2:172.21.15.195:3300/0 conn(0x7fe18c0c9a10 0x7fe18c0ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:35.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.325+0000 7fe190d4d640 1 --2- 172.21.15.107:0/4204014815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe18c151fb0 0x7fe18c0724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:35.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.325+0000 7fe190d4d640 1 -- 172.21.15.107:0/4204014815 >> 172.21.15.107:0/4204014815 conn(0x7fe18c066570 msgr2=0x7fe18c066980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:35.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.325+0000 7fe190d4d640 1 -- 172.21.15.107:0/4204014815 shutdown_connections 2023-12-04T05:39:35.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.325+0000 7fe190d4d640 1 -- 172.21.15.107:0/4204014815 wait complete. 2023-12-04T05:39:35.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.325+0000 7fe190d4d640 1 Processor -- start 2023-12-04T05:39:35.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.325+0000 7fe190d4d640 1 -- start start 2023-12-04T05:39:35.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe190d4d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe18c0c9a10 0x7fe18c157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:35.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe190d4d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe18c0ca990 0x7fe18c06c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:35.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe190d4d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe18c151fb0 0x7fe18c1f6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:35.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe190d4d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c14e940 con 0x7fe18c0c9a10 2023-12-04T05:39:35.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe190d4d640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c06d4c0 con 0x7fe18c151fb0 2023-12-04T05:39:35.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe190d4d640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c145c20 con 0x7fe18c0ca990 2023-12-04T05:39:35.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe18a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe18c0c9a10 0x7fe18c157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:35.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe18a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe18c0c9a10 0x7fe18c157830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36528/0 (socket says 172.21.15.107:36528) 2023-12-04T05:39:35.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe18a59c640 1 -- 172.21.15.107:0/1080411626 learned_addr learned my addr 172.21.15.107:0/1080411626 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:39:35.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe189d9b640 1 --2- 172.21.15.107:0/1080411626 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe18c0ca990 0x7fe18c06c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:35.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe18ad9d640 1 --2- 172.21.15.107:0/1080411626 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe18c151fb0 0x7fe18c1f6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:35.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe18a59c640 1 -- 172.21.15.107:0/1080411626 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe18c0ca990 msgr2=0x7fe18c06c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:35.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe18a59c640 1 --2- 172.21.15.107:0/1080411626 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe18c0ca990 0x7fe18c06c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:35.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe18a59c640 1 -- 172.21.15.107:0/1080411626 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe18c151fb0 msgr2=0x7fe18c1f6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:35.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe18a59c640 1 --2- 172.21.15.107:0/1080411626 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe18c151fb0 0x7fe18c1f6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:35.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe18a59c640 1 -- 172.21.15.107:0/1080411626 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17c00e590 con 0x7fe18c0c9a10 2023-12-04T05:39:35.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe18a59c640 1 --2- 172.21.15.107:0/1080411626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe18c0c9a10 0x7fe18c157830 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe184013780 tx=0x7fe184013c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:35.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe182ffd640 1 -- 172.21.15.107:0/1080411626 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe184013e90 con 0x7fe18c0c9a10 2023-12-04T05:39:35.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe182ffd640 1 -- 172.21.15.107:0/1080411626 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18400c050 con 0x7fe18c0c9a10 2023-12-04T05:39:35.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe190d4d640 1 -- 172.21.15.107:0/1080411626 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe18c1f63c0 con 0x7fe18c0c9a10 2023-12-04T05:39:35.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe182ffd640 1 -- 172.21.15.107:0/1080411626 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe184011c10 con 0x7fe18c0c9a10 2023-12-04T05:39:35.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe190d4d640 1 -- 172.21.15.107:0/1080411626 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe18c06cf00 con 0x7fe18c0c9a10 2023-12-04T05:39:35.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe182ffd640 1 -- 172.21.15.107:0/1080411626 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe184005ca0 con 0x7fe18c0c9a10 2023-12-04T05:39:35.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe182ffd640 1 -- 172.21.15.107:0/1080411626 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe1840167a0 con 0x7fe18c0c9a10 2023-12-04T05:39:35.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.329+0000 7fe190d4d640 1 -- 172.21.15.107:0/1080411626 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe18c067ec0 con 0x7fe18c0c9a10 2023-12-04T05:39:35.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.333+0000 7fe182ffd640 1 -- 172.21.15.107:0/1080411626 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fe18403dce0 con 0x7fe18c0c9a10 2023-12-04T05:39:35.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.393+0000 7fe190d4d640 1 -- 172.21.15.107:0/1080411626 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe18c073070 con 0x7fe18c0c9a10 2023-12-04T05:39:35.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.397+0000 7fe182ffd640 1 -- 172.21.15.107:0/1080411626 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe18402c020 con 0x7fe18c0c9a10 2023-12-04T05:39:35.399 INFO:teuthology.orchestra.run.smithi107.stderr:adjusted tunables profile to default 2023-12-04T05:39:35.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.397+0000 7fe190d4d640 1 -- 172.21.15.107:0/1080411626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe18c0c9a10 msgr2=0x7fe18c157830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:35.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.397+0000 7fe190d4d640 1 --2- 172.21.15.107:0/1080411626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe18c0c9a10 0x7fe18c157830 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe184013780 tx=0x7fe184013c40 comp rx=0 tx=0).stop 2023-12-04T05:39:35.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.397+0000 7fe190d4d640 1 -- 172.21.15.107:0/1080411626 shutdown_connections 2023-12-04T05:39:35.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.397+0000 7fe190d4d640 1 --2- 172.21.15.107:0/1080411626 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe18c151fb0 0x7fe18c1f6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:35.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.397+0000 7fe190d4d640 1 --2- 172.21.15.107:0/1080411626 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe18c0ca990 0x7fe18c06c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:35.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.397+0000 7fe190d4d640 1 --2- 172.21.15.107:0/1080411626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe18c0c9a10 0x7fe18c157830 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:35.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.397+0000 7fe190d4d640 1 -- 172.21.15.107:0/1080411626 >> 172.21.15.107:0/1080411626 conn(0x7fe18c066570 msgr2=0x7fe18c0cb3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:35.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.397+0000 7fe190d4d640 1 -- 172.21.15.107:0/1080411626 shutdown_connections 2023-12-04T05:39:35.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:35.397+0000 7fe190d4d640 1 -- 172.21.15.107:0/1080411626 wait complete. 2023-12-04T05:39:35.418 INFO:tasks.ceph:check_enable_crimson: False 2023-12-04T05:39:35.418 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-04T05:39:35.419 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:39:35.419 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-04T05:39:35.430 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:39:35.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-04T05:39:35.489 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:39:35.489 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-04T05:39:35.541 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:39:35.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-12-04T05:39:35.597 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:39:35.598 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-12-04T05:39:35.611 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:39:35.611 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-12-04T05:39:35.669 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:39:35.669 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-12-04T05:39:35.676 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:35.673+0000 7fdb3fdd0180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-04T05:39:35.721 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-12-04T05:39:35.721 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-12-04T05:39:35.736 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:35.733+0000 7f7cb9017180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-04T05:39:35.747 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 0ab7111c-0ae8-4d41-b6db-e3743614322c 0 2023-12-04T05:39:35.859 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:35.857+0000 7fdb3fdd0180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-04T05:39:35.909 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.905+0000 7f1476898640 1 Processor -- start 2023-12-04T05:39:35.909 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.905+0000 7f1476898640 1 -- start start 2023-12-04T05:39:35.911 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.909+0000 7f1476898640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f147006dc50 0x7f147015b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:35.911 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.909+0000 7f1476898640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1470072840 0x7f1470072c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:35.911 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.909+0000 7f1476898640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14700703b0 con 0x7f1470151f70 2023-12-04T05:39:35.911 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.909+0000 7f1476898640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14700731b0 con 0x7f1470072840 2023-12-04T05:39:35.911 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.909+0000 7f1476898640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1470070c50 con 0x7f147006dc50 2023-12-04T05:39:35.911 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.909+0000 7f146ffff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f1470151f70 0x7f14700663a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52316/0 (socket says 172.21.15.195:52316) 2023-12-04T05:39:35.911 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.909+0000 7f1467fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f147006dc50 0x7f147015b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:35.911 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.909+0000 7f146ffff640 1 -- 172.21.15.195:0/904535098 learned_addr learned my addr 172.21.15.195:0/904535098 (peer_addr_for_me v1:172.21.15.195:0/0) 2023-12-04T05:39:35.911 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.909+0000 7f1474e35640 1 --2- 172.21.15.195:0/904535098 >> v2:172.21.15.107:3300/0 conn(0x7f1470072840 0x7f1470072c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:35.911 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.909+0000 7f1467fff640 1 -- 172.21.15.195:0/904535098 >> v2:172.21.15.107:3300/0 conn(0x7f1470072840 msgr2=0x7f1470072c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:35.911 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.909+0000 7f1467fff640 1 --2- 172.21.15.195:0/904535098 >> v2:172.21.15.107:3300/0 conn(0x7f1470072840 0x7f1470072c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:35.912 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.909+0000 7f1467fff640 1 -- 172.21.15.195:0/904535098 >> v1:172.21.15.195:6789/0 conn(0x7f1470151f70 legacy=0x7f14700663a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:35.912 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.909+0000 7f1467fff640 1 -- 172.21.15.195:0/904535098 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147006e220 con 0x7f147006dc50 2023-12-04T05:39:35.912 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.909+0000 7f146f7fe640 1 -- 172.21.15.195:0/904535098 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299881218 0 0) 0x7f14700703b0 con 0x7f1470151f70 2023-12-04T05:39:35.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.909+0000 7f1467fff640 1 --2- 172.21.15.195:0/904535098 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f147006dc50 0x7f147015b260 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f147006ed70 tx=0x7f145801c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2164ad28abb66cc2 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:35.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.909+0000 7f146f7fe640 1 -- 172.21.15.195:0/904535098 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14580ab660 con 0x7f147006dc50 2023-12-04T05:39:35.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.909+0000 7f146f7fe640 1 -- 172.21.15.195:0/904535098 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14580ab7c0 con 0x7f147006dc50 2023-12-04T05:39:35.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.909+0000 7f146f7fe640 1 -- 172.21.15.195:0/904535098 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14580abb50 con 0x7f147006dc50 2023-12-04T05:39:35.914 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1476898640 1 -- 172.21.15.195:0/904535098 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f147006dc50 msgr2=0x7f147015b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:35.914 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1476898640 1 --2- 172.21.15.195:0/904535098 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f147006dc50 0x7f147015b260 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f147006ed70 tx=0x7f145801c460 comp rx=0 tx=0).stop 2023-12-04T05:39:35.914 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1476898640 1 -- 172.21.15.195:0/904535098 shutdown_connections 2023-12-04T05:39:35.914 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1476898640 1 --2- 172.21.15.195:0/904535098 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f147006dc50 0x7f147015b260 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:35.914 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1476898640 1 --2- 172.21.15.195:0/904535098 >> v2:172.21.15.107:3300/0 conn(0x7f1470072840 0x7f1470072c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:35.914 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1476898640 1 -- 172.21.15.195:0/904535098 >> 172.21.15.195:0/904535098 conn(0x7f147006ada0 msgr2=0x7f14700cbe30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:35.914 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1476898640 1 -- 172.21.15.195:0/904535098 shutdown_connections 2023-12-04T05:39:35.915 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1476898640 1 -- 172.21.15.195:0/904535098 wait complete. 2023-12-04T05:39:35.915 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1476898640 1 Processor -- start 2023-12-04T05:39:35.915 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1476898640 1 -- start start 2023-12-04T05:39:35.917 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1476898640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f147006dc50 0x7f14700cb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:35.917 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1476898640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1470072840 0x7f14701f04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:35.917 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1476898640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1470151f70 0x7f14701f09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:35.917 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1476898640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14701f0050 con 0x7f147006dc50 2023-12-04T05:39:35.917 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1476898640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147013fe50 con 0x7f1470072840 2023-12-04T05:39:35.917 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1476898640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14700c4330 con 0x7f1470151f70 2023-12-04T05:39:35.917 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1467fff640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1470072840 0x7f14701f04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:35.917 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1474e35640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1470151f70 0x7f14701f09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:35.917 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1467fff640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1470072840 0x7f14701f04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44008/0 (socket says 172.21.15.195:44008) 2023-12-04T05:39:35.918 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1467fff640 1 -- 172.21.15.195:0/3542908453 learned_addr learned my addr 172.21.15.195:0/3542908453 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-12-04T05:39:35.918 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f146ffff640 1 --2- 172.21.15.195:0/3542908453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f147006dc50 0x7f14700cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:35.918 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1474e35640 1 -- 172.21.15.195:0/3542908453 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1470072840 msgr2=0x7f14701f04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:35.918 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1474e35640 1 --2- 172.21.15.195:0/3542908453 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1470072840 0x7f14701f04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:35.918 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1474e35640 1 -- 172.21.15.195:0/3542908453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f147006dc50 msgr2=0x7f14700cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:35.918 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1474e35640 1 --2- 172.21.15.195:0/3542908453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f147006dc50 0x7f14700cb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:35.918 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1474e35640 1 -- 172.21.15.195:0/3542908453 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1458009420 con 0x7f1470151f70 2023-12-04T05:39:35.918 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1467fff640 1 --2- 172.21.15.195:0/3542908453 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1470072840 0x7f14701f04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T05:39:35.918 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1474e35640 1 --2- 172.21.15.195:0/3542908453 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1470151f70 0x7f14701f09e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1468007d90 tx=0x7f146800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:35.918 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f146d7fa640 1 -- 172.21.15.195:0/3542908453 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468018070 con 0x7f1470151f70 2023-12-04T05:39:35.918 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f146d7fa640 1 -- 172.21.15.195:0/3542908453 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f146800edd0 con 0x7f1470151f70 2023-12-04T05:39:35.918 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1476898640 1 -- 172.21.15.195:0/3542908453 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14700c45f0 con 0x7f1470151f70 2023-12-04T05:39:35.918 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f146d7fa640 1 -- 172.21.15.195:0/3542908453 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468003bc0 con 0x7f1470151f70 2023-12-04T05:39:35.918 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f1476898640 1 -- 172.21.15.195:0/3542908453 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14701400f0 con 0x7f1470151f70 2023-12-04T05:39:35.919 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f146d7fa640 1 -- 172.21.15.195:0/3542908453 <== mon.2 v2:172.21.15.195:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1468014720 con 0x7f1470151f70 2023-12-04T05:39:35.919 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.913+0000 7f146d7fa640 1 -- 172.21.15.195:0/3542908453 <== mon.2 v2:172.21.15.195:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f146801d940 con 0x7f1470151f70 2023-12-04T05:39:35.919 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.917+0000 7f1476898640 1 -- 172.21.15.195:0/3542908453 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14700710c0 con 0x7f1470151f70 2023-12-04T05:39:35.930 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.929+0000 7f146d7fa640 1 -- 172.21.15.195:0/3542908453 <== mon.2 v2:172.21.15.195:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f14700710c0 con 0x7f1470151f70 2023-12-04T05:39:35.941 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:35.937+0000 7f7cb9017180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-04T05:39:35.999 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:35.997+0000 7f1476898640 1 -- 172.21.15.195:0/3542908453 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "osd new", "uuid": "0ab7111c-0ae8-4d41-b6db-e3743614322c", "id": 0} v 0) v1 -- 0x7f14701f01c0 con 0x7f1470151f70 2023-12-04T05:39:36.009 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.005+0000 7f146d7fa640 1 -- 172.21.15.195:0/3542908453 <== mon.2 v2:172.21.15.195:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0ab7111c-0ae8-4d41-b6db-e3743614322c", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f14701f01c0 con 0x7f1470151f70 2023-12-04T05:39:36.009 INFO:teuthology.orchestra.run.smithi195.stdout:0 2023-12-04T05:39:36.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.009+0000 7f1476898640 1 -- 172.21.15.195:0/3542908453 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1470151f70 msgr2=0x7f14701f09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:36.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.009+0000 7f1476898640 1 --2- 172.21.15.195:0/3542908453 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1470151f70 0x7f14701f09e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1468007d90 tx=0x7f146800e4b0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.009+0000 7f1476898640 1 -- 172.21.15.195:0/3542908453 shutdown_connections 2023-12-04T05:39:36.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.009+0000 7f1476898640 1 --2- 172.21.15.195:0/3542908453 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1470151f70 0x7f14701f09e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.009+0000 7f1476898640 1 --2- 172.21.15.195:0/3542908453 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1470072840 0x7f14701f04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.009+0000 7f1476898640 1 --2- 172.21.15.195:0/3542908453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f147006dc50 0x7f14700cb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.009+0000 7f1476898640 1 -- 172.21.15.195:0/3542908453 >> 172.21.15.195:0/3542908453 conn(0x7f147006ada0 msgr2=0x7f14700cbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:36.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.009+0000 7f1476898640 1 -- 172.21.15.195:0/3542908453 shutdown_connections 2023-12-04T05:39:36.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.009+0000 7f1476898640 1 -- 172.21.15.195:0/3542908453 wait complete. 2023-12-04T05:39:36.027 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new baef7a1f-8026-4fab-9e70-48465271034e 1 2023-12-04T05:39:36.148 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.145+0000 7f42f7541640 1 Processor -- start 2023-12-04T05:39:36.148 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.145+0000 7f42f7541640 1 -- start start 2023-12-04T05:39:36.149 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.145+0000 7f42f7541640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42f0071030 0x7f42f00c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:36.149 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.145+0000 7f42f7541640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f42f00c60e0 0x7f42f006e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:36.149 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.145+0000 7f42f7541640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f0152360 con 0x7f42f0151f80 2023-12-04T05:39:36.149 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.145+0000 7f42f7541640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f006acf0 con 0x7f42f00c60e0 2023-12-04T05:39:36.149 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.145+0000 7f42f7541640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f0071430 con 0x7f42f0071030 2023-12-04T05:39:36.149 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.145+0000 7f42f52dd640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f42f0151f80 0x7f42f006c2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52318/0 (socket says 172.21.15.195:52318) 2023-12-04T05:39:36.149 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.145+0000 7f42f52dd640 1 -- 172.21.15.195:0/96689386 learned_addr learned my addr 172.21.15.195:0/96689386 (peer_addr_for_me v1:172.21.15.195:0/0) 2023-12-04T05:39:36.150 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.145+0000 7f42f4adc640 1 --2- 172.21.15.195:0/96689386 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42f0071030 0x7f42f00c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:36.150 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.145+0000 7f42f5ade640 1 --2- 172.21.15.195:0/96689386 >> v2:172.21.15.195:3300/0 conn(0x7f42f00c60e0 0x7f42f006e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:36.150 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.149+0000 7f42e7fff640 1 -- 172.21.15.195:0/96689386 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3509260413 0 0) 0x7f42f0152360 con 0x7f42f0151f80 2023-12-04T05:39:36.150 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.149+0000 7f42e7fff640 1 -- 172.21.15.195:0/96689386 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42dc003530 con 0x7f42f0151f80 2023-12-04T05:39:36.150 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.149+0000 7f42f4adc640 1 -- 172.21.15.195:0/96689386 >> v2:172.21.15.195:3300/0 conn(0x7f42f00c60e0 msgr2=0x7f42f006e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:36.150 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.149+0000 7f42f4adc640 1 --2- 172.21.15.195:0/96689386 >> v2:172.21.15.195:3300/0 conn(0x7f42f00c60e0 0x7f42f006e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.150 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.149+0000 7f42f4adc640 1 -- 172.21.15.195:0/96689386 >> v1:172.21.15.195:6789/0 conn(0x7f42f0151f80 legacy=0x7f42f006c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:36.150 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.149+0000 7f42f4adc640 1 -- 172.21.15.195:0/96689386 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f0157c90 con 0x7f42f0071030 2023-12-04T05:39:36.152 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.149+0000 7f42f4adc640 1 --2- 172.21.15.195:0/96689386 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42f0071030 0x7f42f00c5ba0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f42e00097c0 tx=0x7f42e001ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f639ed90d159939 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:36.152 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.149+0000 7f42e7fff640 1 -- 172.21.15.195:0/96689386 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e00a8970 con 0x7f42f0071030 2023-12-04T05:39:36.152 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.149+0000 7f42e7fff640 1 -- 172.21.15.195:0/96689386 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e00a8ad0 con 0x7f42f0071030 2023-12-04T05:39:36.152 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.149+0000 7f42e7fff640 1 -- 172.21.15.195:0/96689386 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e00a8e60 con 0x7f42f0071030 2023-12-04T05:39:36.153 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.149+0000 7f42f7541640 1 -- 172.21.15.195:0/96689386 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42f0071030 msgr2=0x7f42f00c5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:36.153 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.149+0000 7f42f7541640 1 --2- 172.21.15.195:0/96689386 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42f0071030 0x7f42f00c5ba0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f42e00097c0 tx=0x7f42e001ec70 comp rx=0 tx=0).stop 2023-12-04T05:39:36.153 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.149+0000 7f42f7541640 1 -- 172.21.15.195:0/96689386 shutdown_connections 2023-12-04T05:39:36.153 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.149+0000 7f42f7541640 1 --2- 172.21.15.195:0/96689386 >> v2:172.21.15.195:3300/0 conn(0x7f42f00c60e0 0x7f42f006e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.153 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.149+0000 7f42f7541640 1 --2- 172.21.15.195:0/96689386 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42f0071030 0x7f42f00c5ba0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.153 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.149+0000 7f42f7541640 1 -- 172.21.15.195:0/96689386 >> 172.21.15.195:0/96689386 conn(0x7f42f00cc490 msgr2=0x7f42f00cc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:36.153 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.149+0000 7f42f7541640 1 -- 172.21.15.195:0/96689386 shutdown_connections 2023-12-04T05:39:36.153 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.149+0000 7f42f7541640 1 -- 172.21.15.195:0/96689386 wait complete. 2023-12-04T05:39:36.154 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f7541640 1 Processor -- start 2023-12-04T05:39:36.154 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f7541640 1 -- start start 2023-12-04T05:39:36.156 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f7541640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42f0071030 0x7f42f0155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:36.156 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f7541640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42f00c60e0 0x7f42f01edcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:36.156 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f7541640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f42f0151f80 0x7f42f00cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:36.156 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f7541640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f0152a70 con 0x7f42f00c60e0 2023-12-04T05:39:36.156 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f7541640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f01524f0 con 0x7f42f0151f80 2023-12-04T05:39:36.156 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f7541640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f01ed500 con 0x7f42f0071030 2023-12-04T05:39:36.156 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f52dd640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42f0071030 0x7f42f0155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:36.156 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f52dd640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42f0071030 0x7f42f0155c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.195:54000/0 (socket says 172.21.15.195:54000) 2023-12-04T05:39:36.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f52dd640 1 -- 172.21.15.195:0/3452203560 learned_addr learned my addr 172.21.15.195:0/3452203560 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-12-04T05:39:36.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f5ade640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f42f0151f80 0x7f42f00cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:36.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f4adc640 1 --2- 172.21.15.195:0/3452203560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42f00c60e0 0x7f42f01edcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:36.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f52dd640 1 -- 172.21.15.195:0/3452203560 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f42f0151f80 msgr2=0x7f42f00cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:36.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f52dd640 1 --2- 172.21.15.195:0/3452203560 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f42f0151f80 0x7f42f00cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f52dd640 1 -- 172.21.15.195:0/3452203560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42f00c60e0 msgr2=0x7f42f01edcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:36.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f52dd640 1 --2- 172.21.15.195:0/3452203560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42f00c60e0 0x7f42f01edcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f52dd640 1 -- 172.21.15.195:0/3452203560 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e0009420 con 0x7f42f0071030 2023-12-04T05:39:36.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f4adc640 1 --2- 172.21.15.195:0/3452203560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42f00c60e0 0x7f42f01edcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:39:36.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f5ade640 1 --2- 172.21.15.195:0/3452203560 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f42f0151f80 0x7f42f00cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T05:39:36.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f52dd640 1 --2- 172.21.15.195:0/3452203560 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42f0071030 0x7f42f0155c50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f42e800c450 tx=0x7f42e800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:36.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42e5ffb640 1 -- 172.21.15.195:0/3452203560 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e8005cf0 con 0x7f42f0071030 2023-12-04T05:39:36.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42e5ffb640 1 -- 172.21.15.195:0/3452203560 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e8005e50 con 0x7f42f0071030 2023-12-04T05:39:36.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f7541640 1 -- 172.21.15.195:0/3452203560 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f0152790 con 0x7f42f0071030 2023-12-04T05:39:36.158 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42e5ffb640 1 -- 172.21.15.195:0/3452203560 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e8007270 con 0x7f42f0071030 2023-12-04T05:39:36.158 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f7541640 1 -- 172.21.15.195:0/3452203560 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f014c8f0 con 0x7f42f0071030 2023-12-04T05:39:36.158 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42e5ffb640 1 -- 172.21.15.195:0/3452203560 <== mon.2 v2:172.21.15.195:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f42e801c440 con 0x7f42f0071030 2023-12-04T05:39:36.158 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42e5ffb640 1 -- 172.21.15.195:0/3452203560 <== mon.2 v2:172.21.15.195:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f42e80127f0 con 0x7f42f0071030 2023-12-04T05:39:36.158 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.153+0000 7f42f7541640 1 -- 172.21.15.195:0/3452203560 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f01edb10 con 0x7f42f0071030 2023-12-04T05:39:36.158 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.157+0000 7f42e5ffb640 1 -- 172.21.15.195:0/3452203560 <== mon.2 v2:172.21.15.195:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f42e8021060 con 0x7f42f0071030 2023-12-04T05:39:36.227 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.225+0000 7f42f7541640 1 -- 172.21.15.195:0/3452203560 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "osd new", "uuid": "baef7a1f-8026-4fab-9e70-48465271034e", "id": 1} v 0) v1 -- 0x7f42f0067ec0 con 0x7f42f0071030 2023-12-04T05:39:36.235 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.233+0000 7f42e5ffb640 1 -- 172.21.15.195:0/3452203560 <== mon.2 v2:172.21.15.195:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "baef7a1f-8026-4fab-9e70-48465271034e", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f42e8017070 con 0x7f42f0071030 2023-12-04T05:39:36.235 INFO:teuthology.orchestra.run.smithi195.stdout:1 2023-12-04T05:39:36.236 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.233+0000 7f42f7541640 1 -- 172.21.15.195:0/3452203560 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42f0071030 msgr2=0x7f42f0155c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:36.236 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.233+0000 7f42f7541640 1 --2- 172.21.15.195:0/3452203560 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42f0071030 0x7f42f0155c50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f42e800c450 tx=0x7f42e800c910 comp rx=0 tx=0).stop 2023-12-04T05:39:36.237 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.233+0000 7f42f7541640 1 -- 172.21.15.195:0/3452203560 shutdown_connections 2023-12-04T05:39:36.237 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.233+0000 7f42f7541640 1 --2- 172.21.15.195:0/3452203560 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f42f0151f80 0x7f42f00cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.237 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.233+0000 7f42f7541640 1 --2- 172.21.15.195:0/3452203560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42f00c60e0 0x7f42f01edcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.237 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.233+0000 7f42f7541640 1 --2- 172.21.15.195:0/3452203560 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f42f0071030 0x7f42f0155c50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.237 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.233+0000 7f42f7541640 1 -- 172.21.15.195:0/3452203560 >> 172.21.15.195:0/3452203560 conn(0x7f42f00cc490 msgr2=0x7f42f006e250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:36.237 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.233+0000 7f42f7541640 1 -- 172.21.15.195:0/3452203560 shutdown_connections 2023-12-04T05:39:36.237 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.233+0000 7f42f7541640 1 -- 172.21.15.195:0/3452203560 wait complete. 2023-12-04T05:39:36.252 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 90e8cf24-7b54-484a-9d85-b44a9d3dfeab 2 2023-12-04T05:39:36.366 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.365+0000 7f3b08126640 1 Processor -- start 2023-12-04T05:39:36.367 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.365+0000 7f3b08126640 1 -- start start 2023-12-04T05:39:36.368 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.365+0000 7f3b08126640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3b0006e730 0x7f3b00070ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:36.368 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.365+0000 7f3b08126640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3b00072590 0x7f3b00072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:36.368 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.365+0000 7f3b08126640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b00072100 con 0x7f3b0006e350 2023-12-04T05:39:36.368 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.365+0000 7f3b08126640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b000715b0 con 0x7f3b0006e730 2023-12-04T05:39:36.368 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.365+0000 7f3b08126640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b0006f190 con 0x7f3b00072590 2023-12-04T05:39:36.369 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.365+0000 7f3b066c3640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3b00072590 0x7f3b00072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:36.369 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.365+0000 7f3b066c3640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3b00072590 0x7f3b00072a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44022/0 (socket says 172.21.15.195:44022) 2023-12-04T05:39:36.369 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.365+0000 7f3b066c3640 1 -- 172.21.15.195:0/3883464037 learned_addr learned my addr 172.21.15.195:0/3883464037 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-12-04T05:39:36.369 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.365+0000 7f3b056c1640 1 --2- 172.21.15.195:0/3883464037 >> v2:172.21.15.107:3300/0 conn(0x7f3b0006e730 0x7f3b00070ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:36.369 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.365+0000 7f3b066c3640 1 -- 172.21.15.195:0/3883464037 >> v2:172.21.15.107:3300/0 conn(0x7f3b0006e730 msgr2=0x7f3b00070ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:36.369 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.365+0000 7f3b066c3640 1 --2- 172.21.15.195:0/3883464037 >> v2:172.21.15.107:3300/0 conn(0x7f3b0006e730 0x7f3b00070ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.369 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.365+0000 7f3b066c3640 1 -- 172.21.15.195:0/3883464037 >> v1:172.21.15.107:6789/0 conn(0x7f3b0006e350 legacy=0x7f3b001576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:36.369 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.365+0000 7f3b066c3640 1 -- 172.21.15.195:0/3883464037 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0006f470 con 0x7f3b00072590 2023-12-04T05:39:36.370 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b066c3640 1 --2- 172.21.15.195:0/3883464037 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3b00072590 0x7f3b00072a00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3b0015b2c0 tx=0x7f3af801f1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1d7dda53ed433e5 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:36.370 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b04ec0640 1 -- 172.21.15.195:0/3883464037 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af80af400 con 0x7f3b00072590 2023-12-04T05:39:36.370 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b04ec0640 1 -- 172.21.15.195:0/3883464037 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af80ab050 con 0x7f3b00072590 2023-12-04T05:39:36.370 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b04ec0640 1 -- 172.21.15.195:0/3883464037 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af80af780 con 0x7f3b00072590 2023-12-04T05:39:36.371 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b08126640 1 -- 172.21.15.195:0/3883464037 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3b00072590 msgr2=0x7f3b00072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:36.371 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b08126640 1 --2- 172.21.15.195:0/3883464037 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3b00072590 0x7f3b00072a00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3b0015b2c0 tx=0x7f3af801f1c0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.371 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b08126640 1 -- 172.21.15.195:0/3883464037 shutdown_connections 2023-12-04T05:39:36.371 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b08126640 1 --2- 172.21.15.195:0/3883464037 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3b00072590 0x7f3b00072a00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.371 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b08126640 1 --2- 172.21.15.195:0/3883464037 >> v2:172.21.15.107:3300/0 conn(0x7f3b0006e730 0x7f3b00070ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.372 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b08126640 1 -- 172.21.15.195:0/3883464037 >> 172.21.15.195:0/3883464037 conn(0x7f3b000cc440 msgr2=0x7f3b000cc850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:36.372 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b08126640 1 -- 172.21.15.195:0/3883464037 shutdown_connections 2023-12-04T05:39:36.372 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b08126640 1 -- 172.21.15.195:0/3883464037 wait complete. 2023-12-04T05:39:36.372 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b08126640 1 Processor -- start 2023-12-04T05:39:36.372 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b08126640 1 -- start start 2023-12-04T05:39:36.375 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b08126640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3b0006e350 0x7f3b0015ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:36.375 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b08126640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b0006e730 0x7f3b00148650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:36.376 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b08126640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3b00072590 0x7f3b00156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:36.376 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b08126640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b001ee300 con 0x7f3b0006e730 2023-12-04T05:39:36.376 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b08126640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b001ee830 con 0x7f3b00072590 2023-12-04T05:39:36.376 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b08126640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b001ee550 con 0x7f3b0006e350 2023-12-04T05:39:36.376 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b066c3640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3b00072590 0x7f3b00156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:36.376 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b05ec2640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3b0006e350 0x7f3b0015ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:36.377 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b066c3640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3b00072590 0x7f3b00156eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44030/0 (socket says 172.21.15.195:44030) 2023-12-04T05:39:36.377 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b066c3640 1 -- 172.21.15.195:0/1285378258 learned_addr learned my addr 172.21.15.195:0/1285378258 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-12-04T05:39:36.377 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b056c1640 1 --2- 172.21.15.195:0/1285378258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b0006e730 0x7f3b00148650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:36.377 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b05ec2640 1 -- 172.21.15.195:0/1285378258 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3b00072590 msgr2=0x7f3b00156eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:36.377 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.369+0000 7f3b05ec2640 1 --2- 172.21.15.195:0/1285378258 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3b00072590 0x7f3b00156eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.377 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.373+0000 7f3b05ec2640 1 -- 172.21.15.195:0/1285378258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b0006e730 msgr2=0x7f3b00148650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:36.377 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.373+0000 7f3b05ec2640 1 --2- 172.21.15.195:0/1285378258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b0006e730 0x7f3b00148650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.378 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.373+0000 7f3b05ec2640 1 -- 172.21.15.195:0/1285378258 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af8009b30 con 0x7f3b0006e350 2023-12-04T05:39:36.378 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.373+0000 7f3b05ec2640 1 --2- 172.21.15.195:0/1285378258 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3b0006e350 0x7f3b0015ab50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3afc000f70 tx=0x7f3afc00c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:36.378 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.373+0000 7f3af67fc640 1 -- 172.21.15.195:0/1285378258 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc0052d0 con 0x7f3b0006e350 2023-12-04T05:39:36.378 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.373+0000 7f3af67fc640 1 -- 172.21.15.195:0/1285378258 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3afc005430 con 0x7f3b0006e350 2023-12-04T05:39:36.378 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.373+0000 7f3af67fc640 1 -- 172.21.15.195:0/1285378258 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc003840 con 0x7f3b0006e350 2023-12-04T05:39:36.378 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.373+0000 7f3b08126640 1 -- 172.21.15.195:0/1285378258 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b00147b60 con 0x7f3b0006e350 2023-12-04T05:39:36.378 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.373+0000 7f3b08126640 1 -- 172.21.15.195:0/1285378258 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b001573f0 con 0x7f3b0006e350 2023-12-04T05:39:36.379 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.373+0000 7f3af67fc640 1 -- 172.21.15.195:0/1285378258 <== mon.2 v2:172.21.15.195:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3afc003ca0 con 0x7f3b0006e350 2023-12-04T05:39:36.379 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.373+0000 7f3af67fc640 1 -- 172.21.15.195:0/1285378258 <== mon.2 v2:172.21.15.195:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f3afc012840 con 0x7f3b0006e350 2023-12-04T05:39:36.379 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.373+0000 7f3b08126640 1 -- 172.21.15.195:0/1285378258 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b00067ec0 con 0x7f3b0006e350 2023-12-04T05:39:36.379 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.373+0000 7f3af67fc640 1 -- 172.21.15.195:0/1285378258 <== mon.2 v2:172.21.15.195:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f3afc017070 con 0x7f3b0006e350 2023-12-04T05:39:36.437 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:36.433+0000 7fdb3fdd0180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-04T05:39:36.441 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.437+0000 7f3b08126640 1 -- 172.21.15.195:0/1285378258 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "osd new", "uuid": "90e8cf24-7b54-484a-9d85-b44a9d3dfeab", "id": 2} v 0) v1 -- 0x7f3b0014ee40 con 0x7f3b0006e350 2023-12-04T05:39:36.448 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.445+0000 7f3af67fc640 1 -- 172.21.15.195:0/1285378258 <== mon.2 v2:172.21.15.195:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "90e8cf24-7b54-484a-9d85-b44a9d3dfeab", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3afc0124a0 con 0x7f3b0006e350 2023-12-04T05:39:36.448 INFO:teuthology.orchestra.run.smithi195.stdout:2 2023-12-04T05:39:36.449 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.445+0000 7f3b08126640 1 -- 172.21.15.195:0/1285378258 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3b0006e350 msgr2=0x7f3b0015ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:36.449 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.445+0000 7f3b08126640 1 --2- 172.21.15.195:0/1285378258 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3b0006e350 0x7f3b0015ab50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3afc000f70 tx=0x7f3afc00c820 comp rx=0 tx=0).stop 2023-12-04T05:39:36.449 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.445+0000 7f3b08126640 1 -- 172.21.15.195:0/1285378258 shutdown_connections 2023-12-04T05:39:36.449 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.445+0000 7f3b08126640 1 --2- 172.21.15.195:0/1285378258 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3b00072590 0x7f3b00156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.449 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.445+0000 7f3b08126640 1 --2- 172.21.15.195:0/1285378258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b0006e730 0x7f3b00148650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.450 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.445+0000 7f3b08126640 1 --2- 172.21.15.195:0/1285378258 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3b0006e350 0x7f3b0015ab50 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.450 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.445+0000 7f3b08126640 1 -- 172.21.15.195:0/1285378258 >> 172.21.15.195:0/1285378258 conn(0x7f3b000cc440 msgr2=0x7f3b000729a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:36.450 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.445+0000 7f3b08126640 1 -- 172.21.15.195:0/1285378258 shutdown_connections 2023-12-04T05:39:36.450 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.445+0000 7f3b08126640 1 -- 172.21.15.195:0/1285378258 wait complete. 2023-12-04T05:39:36.464 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 54658731-2b9a-4d5e-ab13-7c1be7d34e7e 3 2023-12-04T05:39:36.492 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:36.489+0000 7fdb3fdd0180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-04T05:39:36.543 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:36.541+0000 7fdb3fdd0180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-04T05:39:36.564 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:36.561+0000 7f7cb9017180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-04T05:39:36.584 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.581+0000 7fe735167640 1 Processor -- start 2023-12-04T05:39:36.584 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.581+0000 7fe735167640 1 -- start start 2023-12-04T05:39:36.586 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.581+0000 7fe735167640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe7300c7910 0x7fe7300c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:36.586 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.581+0000 7fe735167640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe7300c90d0 0x7fe73015d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:36.586 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.581+0000 7fe735167640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73006adf0 con 0x7fe730154080 2023-12-04T05:39:36.586 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.581+0000 7fe735167640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe730072810 con 0x7fe7300c90d0 2023-12-04T05:39:36.586 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.581+0000 7fe735167640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe730159da0 con 0x7fe7300c7910 2023-12-04T05:39:36.586 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.585+0000 7fe72ed9d640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe730154080 0x7fe73006a5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52334/0 (socket says 172.21.15.195:52334) 2023-12-04T05:39:36.586 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.585+0000 7fe72ed9d640 1 -- 172.21.15.195:0/2622245080 learned_addr learned my addr 172.21.15.195:0/2622245080 (peer_addr_for_me v1:172.21.15.195:0/0) 2023-12-04T05:39:36.586 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.585+0000 7fe72e59c640 1 --2- 172.21.15.195:0/2622245080 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe7300c7910 0x7fe7300c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:36.587 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.585+0000 7fe72f59e640 1 --2- 172.21.15.195:0/2622245080 >> v2:172.21.15.107:3300/0 conn(0x7fe7300c90d0 0x7fe73015d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:36.587 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.585+0000 7fe72dd9b640 1 -- 172.21.15.195:0/2622245080 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776698137 0 0) 0x7fe73006adf0 con 0x7fe730154080 2023-12-04T05:39:36.587 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.585+0000 7fe72e59c640 1 -- 172.21.15.195:0/2622245080 >> v2:172.21.15.107:3300/0 conn(0x7fe7300c90d0 msgr2=0x7fe73015d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:36.587 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.585+0000 7fe72e59c640 1 --2- 172.21.15.195:0/2622245080 >> v2:172.21.15.107:3300/0 conn(0x7fe7300c90d0 0x7fe73015d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.587 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.585+0000 7fe72e59c640 1 -- 172.21.15.195:0/2622245080 >> v1:172.21.15.195:6789/0 conn(0x7fe730154080 legacy=0x7fe73006a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:36.587 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.585+0000 7fe72e59c640 1 -- 172.21.15.195:0/2622245080 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7300c9860 con 0x7fe7300c7910 2023-12-04T05:39:36.589 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.585+0000 7fe72e59c640 1 --2- 172.21.15.195:0/2622245080 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe7300c7910 0x7fe7300c8b90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe730159ee0 tx=0x7fe72401c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc02f7dfddb71b63 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:36.589 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.585+0000 7fe72dd9b640 1 -- 172.21.15.195:0/2622245080 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7240ab600 con 0x7fe7300c7910 2023-12-04T05:39:36.589 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.585+0000 7fe72dd9b640 1 -- 172.21.15.195:0/2622245080 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7240a7050 con 0x7fe7300c7910 2023-12-04T05:39:36.589 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.585+0000 7fe72dd9b640 1 -- 172.21.15.195:0/2622245080 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7240ab980 con 0x7fe7300c7910 2023-12-04T05:39:36.590 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.585+0000 7fe735167640 1 -- 172.21.15.195:0/2622245080 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe7300c7910 msgr2=0x7fe7300c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:36.590 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.585+0000 7fe735167640 1 --2- 172.21.15.195:0/2622245080 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe7300c7910 0x7fe7300c8b90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe730159ee0 tx=0x7fe72401c480 comp rx=0 tx=0).stop 2023-12-04T05:39:36.590 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe735167640 1 -- 172.21.15.195:0/2622245080 shutdown_connections 2023-12-04T05:39:36.590 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe735167640 1 --2- 172.21.15.195:0/2622245080 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe7300c7910 0x7fe7300c8b90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.590 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe735167640 1 --2- 172.21.15.195:0/2622245080 >> v2:172.21.15.107:3300/0 conn(0x7fe7300c90d0 0x7fe73015d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.590 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe735167640 1 -- 172.21.15.195:0/2622245080 >> 172.21.15.195:0/2622245080 conn(0x7fe7300cc500 msgr2=0x7fe7300cc910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:36.590 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe735167640 1 -- 172.21.15.195:0/2622245080 shutdown_connections 2023-12-04T05:39:36.590 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe735167640 1 -- 172.21.15.195:0/2622245080 wait complete. 2023-12-04T05:39:36.591 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe735167640 1 Processor -- start 2023-12-04T05:39:36.591 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe735167640 1 -- start start 2023-12-04T05:39:36.594 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe735167640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe7300c7910 0x7fe73014e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:36.594 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe735167640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7300c90d0 0x7fe7301f6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:36.594 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe735167640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe730154080 0x7fe7300cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:36.594 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe735167640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73014d690 con 0x7fe7300c90d0 2023-12-04T05:39:36.594 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe735167640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7301f58d0 con 0x7fe730154080 2023-12-04T05:39:36.594 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe735167640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7301f31a0 con 0x7fe7300c7910 2023-12-04T05:39:36.594 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe72ed9d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe7300c7910 0x7fe73014e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:36.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe72f59e640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe730154080 0x7fe7300cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:36.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe72ed9d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe7300c7910 0x7fe73014e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.195:54018/0 (socket says 172.21.15.195:54018) 2023-12-04T05:39:36.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe72ed9d640 1 -- 172.21.15.195:0/1213278609 learned_addr learned my addr 172.21.15.195:0/1213278609 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-12-04T05:39:36.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe72e59c640 1 --2- 172.21.15.195:0/1213278609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7300c90d0 0x7fe7301f6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:36.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe72ed9d640 1 -- 172.21.15.195:0/1213278609 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe730154080 msgr2=0x7fe7300cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:36.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe72ed9d640 1 --2- 172.21.15.195:0/1213278609 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe730154080 0x7fe7300cb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe72ed9d640 1 -- 172.21.15.195:0/1213278609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7300c90d0 msgr2=0x7fe7301f6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:36.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe72ed9d640 1 --2- 172.21.15.195:0/1213278609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7300c90d0 0x7fe7301f6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe72ed9d640 1 -- 172.21.15.195:0/1213278609 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe724009420 con 0x7fe7300c7910 2023-12-04T05:39:36.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe72e59c640 1 --2- 172.21.15.195:0/1213278609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7300c90d0 0x7fe7301f6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:39:36.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe72ed9d640 1 --2- 172.21.15.195:0/1213278609 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe7300c7910 0x7fe73014e330 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe72800c7f0 tx=0x7fe72800ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:36.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe7177fe640 1 -- 172.21.15.195:0/1213278609 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72800a640 con 0x7fe7300c7910 2023-12-04T05:39:36.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe7177fe640 1 -- 172.21.15.195:0/1213278609 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72800a7c0 con 0x7fe7300c7910 2023-12-04T05:39:36.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe7177fe640 1 -- 172.21.15.195:0/1213278609 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72800ab70 con 0x7fe7300c7910 2023-12-04T05:39:36.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe735167640 1 -- 172.21.15.195:0/1213278609 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe73014ae50 con 0x7fe7300c7910 2023-12-04T05:39:36.596 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.589+0000 7fe735167640 1 -- 172.21.15.195:0/1213278609 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7301f5b20 con 0x7fe7300c7910 2023-12-04T05:39:36.596 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.593+0000 7fe7177fe640 1 -- 172.21.15.195:0/1213278609 <== mon.2 v2:172.21.15.195:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe7280070d0 con 0x7fe7300c7910 2023-12-04T05:39:36.596 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.593+0000 7fe7177fe640 1 -- 172.21.15.195:0/1213278609 <== mon.2 v2:172.21.15.195:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fe728012930 con 0x7fe7300c7910 2023-12-04T05:39:36.596 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.593+0000 7fe735167640 1 -- 172.21.15.195:0/1213278609 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe730067ec0 con 0x7fe7300c7910 2023-12-04T05:39:36.596 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.593+0000 7fe7177fe640 1 -- 172.21.15.195:0/1213278609 <== mon.2 v2:172.21.15.195:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fe728018070 con 0x7fe7300c7910 2023-12-04T05:39:36.620 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:36.617+0000 7fdb3fdd0180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-04T05:39:36.622 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:36.621+0000 7f7cb9017180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-04T05:39:36.665 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.661+0000 7fe735167640 1 -- 172.21.15.195:0/1213278609 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "osd new", "uuid": "54658731-2b9a-4d5e-ab13-7c1be7d34e7e", "id": 3} v 0) v1 -- 0x7fe7300ccb30 con 0x7fe7300c7910 2023-12-04T05:39:36.671 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:36.669+0000 7fdb3fdd0180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-04T05:39:36.671 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.669+0000 7fe7177fe640 1 -- 172.21.15.195:0/1213278609 <== mon.2 v2:172.21.15.195:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "54658731-2b9a-4d5e-ab13-7c1be7d34e7e", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe728012d30 con 0x7fe7300c7910 2023-12-04T05:39:36.671 INFO:teuthology.orchestra.run.smithi195.stdout:3 2023-12-04T05:39:36.672 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.669+0000 7fe735167640 1 -- 172.21.15.195:0/1213278609 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe7300c7910 msgr2=0x7fe73014e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:36.673 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.669+0000 7fe735167640 1 --2- 172.21.15.195:0/1213278609 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe7300c7910 0x7fe73014e330 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe72800c7f0 tx=0x7fe72800ccb0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.673 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.669+0000 7fe735167640 1 -- 172.21.15.195:0/1213278609 shutdown_connections 2023-12-04T05:39:36.673 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.669+0000 7fe735167640 1 --2- 172.21.15.195:0/1213278609 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe730154080 0x7fe7300cb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.673 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.669+0000 7fe735167640 1 --2- 172.21.15.195:0/1213278609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7300c90d0 0x7fe7301f6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.673 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.669+0000 7fe735167640 1 --2- 172.21.15.195:0/1213278609 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe7300c7910 0x7fe73014e330 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.673 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.669+0000 7fe735167640 1 -- 172.21.15.195:0/1213278609 >> 172.21.15.195:0/1213278609 conn(0x7fe7300cc500 msgr2=0x7fe73014a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:36.673 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.669+0000 7fe735167640 1 -- 172.21.15.195:0/1213278609 shutdown_connections 2023-12-04T05:39:36.673 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.669+0000 7fe735167640 1 -- 172.21.15.195:0/1213278609 wait complete. 2023-12-04T05:39:36.676 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:36.673+0000 7f7cb9017180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-04T05:39:36.688 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 9a9dea13-8afe-485d-aa43-45eda691179b 4 2023-12-04T05:39:36.747 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:36.745+0000 7fdb3fdd0180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-04T05:39:36.756 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:36.753+0000 7f7cb9017180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-04T05:39:36.798 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:36.793+0000 7fdb3fdd0180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-04T05:39:36.802 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.801+0000 7f7c13646640 1 Processor -- start 2023-12-04T05:39:36.802 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.801+0000 7f7c13646640 1 -- start start 2023-12-04T05:39:36.804 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.801+0000 7f7c13646640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7c0c153e00 0x7f7c0c0c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:36.804 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.801+0000 7f7c13646640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0c1541e0 con 0x7f7c0c066450 2023-12-04T05:39:36.804 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.801+0000 7f7c13646640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0c0c84d0 con 0x7f7c0c15d450 2023-12-04T05:39:36.804 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.801+0000 7f7c13646640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c0c73d0 con 0x7f7c0c153e00 2023-12-04T05:39:36.804 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.801+0000 7f7c113e2640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7c0c153e00 0x7f7c0c0c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:36.804 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.801+0000 7f7c11be3640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f7c0c15d450 0x7f7c0c15d8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55264/0 (socket says 172.21.15.195:55264) 2023-12-04T05:39:36.804 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.801+0000 7f7c11be3640 1 -- 172.21.15.195:0/2645006230 learned_addr learned my addr 172.21.15.195:0/2645006230 (peer_addr_for_me v1:172.21.15.195:0/0) 2023-12-04T05:39:36.804 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.801+0000 7f7c113e2640 1 -- 172.21.15.195:0/2645006230 >> v1:172.21.15.195:6789/0 conn(0x7f7c0c15d450 legacy=0x7f7c0c15d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:36.804 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.801+0000 7f7c113e2640 1 -- 172.21.15.195:0/2645006230 >> v1:172.21.15.107:6789/0 conn(0x7f7c0c066450 legacy=0x7f7c0c06dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:36.804 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.801+0000 7f7c03fff640 1 -- 172.21.15.195:0/2645006230 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3635053096 0 0) 0x7f7c0c0c84d0 con 0x7f7c0c15d450 2023-12-04T05:39:36.805 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.801+0000 7f7c113e2640 1 -- 172.21.15.195:0/2645006230 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0c0c8740 con 0x7f7c0c153e00 2023-12-04T05:39:36.806 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c113e2640 1 --2- 172.21.15.195:0/2645006230 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7c0c153e00 0x7f7c0c0c7f90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7c080097f0 tx=0x7f7c0801ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4237d023b3d3031 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:36.806 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c03fff640 1 -- 172.21.15.195:0/2645006230 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c080a89b0 con 0x7f7c0c153e00 2023-12-04T05:39:36.806 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c03fff640 1 -- 172.21.15.195:0/2645006230 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c080a8b10 con 0x7f7c0c153e00 2023-12-04T05:39:36.806 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c03fff640 1 -- 172.21.15.195:0/2645006230 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c080a8ea0 con 0x7f7c0c153e00 2023-12-04T05:39:36.807 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c13646640 1 -- 172.21.15.195:0/2645006230 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7c0c153e00 msgr2=0x7f7c0c0c7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:36.807 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c13646640 1 --2- 172.21.15.195:0/2645006230 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7c0c153e00 0x7f7c0c0c7f90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7c080097f0 tx=0x7f7c0801ecb0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.807 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c13646640 1 -- 172.21.15.195:0/2645006230 shutdown_connections 2023-12-04T05:39:36.807 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c13646640 1 --2- 172.21.15.195:0/2645006230 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7c0c153e00 0x7f7c0c0c7f90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.807 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c13646640 1 -- 172.21.15.195:0/2645006230 >> 172.21.15.195:0/2645006230 conn(0x7f7c0c06ada0 msgr2=0x7f7c0c0cbe30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:36.807 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c13646640 1 -- 172.21.15.195:0/2645006230 shutdown_connections 2023-12-04T05:39:36.807 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c13646640 1 -- 172.21.15.195:0/2645006230 wait complete. 2023-12-04T05:39:36.808 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c13646640 1 Processor -- start 2023-12-04T05:39:36.808 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c13646640 1 -- start start 2023-12-04T05:39:36.810 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c13646640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c0c066450 0x7f7c0c1f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:36.810 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c13646640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7c0c153e00 0x7f7c0c070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:36.810 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c13646640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7c0c15d450 0x7f7c0c1e59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:36.810 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c13646640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c1e5880 con 0x7f7c0c066450 2023-12-04T05:39:36.810 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c13646640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c1f4230 con 0x7f7c0c15d450 2023-12-04T05:39:36.810 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c13646640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c1e4fe0 con 0x7f7c0c153e00 2023-12-04T05:39:36.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c10be1640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7c0c153e00 0x7f7c0c070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:36.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c11be3640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7c0c15d450 0x7f7c0c1e59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:36.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c10be1640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7c0c153e00 0x7f7c0c070790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.195:39604/0 (socket says 172.21.15.195:39604) 2023-12-04T05:39:36.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c10be1640 1 -- 172.21.15.195:0/152400351 learned_addr learned my addr 172.21.15.195:0/152400351 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-12-04T05:39:36.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c113e2640 1 --2- 172.21.15.195:0/152400351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c0c066450 0x7f7c0c1f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:36.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c11be3640 1 -- 172.21.15.195:0/152400351 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7c0c153e00 msgr2=0x7f7c0c070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:36.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c11be3640 1 --2- 172.21.15.195:0/152400351 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7c0c153e00 0x7f7c0c070790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c11be3640 1 -- 172.21.15.195:0/152400351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c0c066450 msgr2=0x7f7c0c1f3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:36.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c11be3640 1 --2- 172.21.15.195:0/152400351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c0c066450 0x7f7c0c1f3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c11be3640 1 -- 172.21.15.195:0/152400351 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c08009420 con 0x7f7c0c15d450 2023-12-04T05:39:36.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c10be1640 1 --2- 172.21.15.195:0/152400351 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7c0c153e00 0x7f7c0c070790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T05:39:36.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c113e2640 1 --2- 172.21.15.195:0/152400351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c0c066450 0x7f7c0c1f3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:39:36.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c11be3640 1 --2- 172.21.15.195:0/152400351 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7c0c15d450 0x7f7c0c1e59e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7c0400c780 tx=0x7f7c0400cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:36.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c01ffb640 1 -- 172.21.15.195:0/152400351 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0400ce30 con 0x7f7c0c15d450 2023-12-04T05:39:36.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c01ffb640 1 -- 172.21.15.195:0/152400351 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0400a640 con 0x7f7c0c15d450 2023-12-04T05:39:36.812 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c13646640 1 -- 172.21.15.195:0/152400351 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c0c1e52c0 con 0x7f7c0c15d450 2023-12-04T05:39:36.812 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c01ffb640 1 -- 172.21.15.195:0/152400351 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0400a9f0 con 0x7f7c0c15d450 2023-12-04T05:39:36.812 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.805+0000 7f7c13646640 1 -- 172.21.15.195:0/152400351 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c0c1f34f0 con 0x7f7c0c15d450 2023-12-04T05:39:36.812 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.809+0000 7f7c01ffb640 1 -- 172.21.15.195:0/152400351 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7c040070d0 con 0x7f7c0c15d450 2023-12-04T05:39:36.812 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.809+0000 7f7c01ffb640 1 -- 172.21.15.195:0/152400351 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f7c04012910 con 0x7f7c0c15d450 2023-12-04T05:39:36.812 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.809+0000 7f7c13646640 1 -- 172.21.15.195:0/152400351 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c0c1f3350 con 0x7f7c0c15d450 2023-12-04T05:39:36.812 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:36.809+0000 7f7cb9017180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-04T05:39:36.812 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.809+0000 7f7c01ffb640 1 -- 172.21.15.195:0/152400351 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f7c0c1f3350 con 0x7f7c0c15d450 2023-12-04T05:39:36.876 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.873+0000 7f7c13646640 1 -- 172.21.15.195:0/152400351 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9a9dea13-8afe-485d-aa43-45eda691179b", "id": 4} v 0) v1 -- 0x7f7c0c067ec0 con 0x7f7c0c15d450 2023-12-04T05:39:36.884 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.881+0000 7f7c01ffb640 1 -- 172.21.15.195:0/152400351 <== mon.1 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9a9dea13-8afe-485d-aa43-45eda691179b", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7c04018070 con 0x7f7c0c15d450 2023-12-04T05:39:36.885 INFO:teuthology.orchestra.run.smithi195.stdout:4 2023-12-04T05:39:36.886 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.881+0000 7f7c13646640 1 -- 172.21.15.195:0/152400351 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7c0c15d450 msgr2=0x7f7c0c1e59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:36.886 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.881+0000 7f7c13646640 1 --2- 172.21.15.195:0/152400351 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7c0c15d450 0x7f7c0c1e59e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7c0400c780 tx=0x7f7c0400cc40 comp rx=0 tx=0).stop 2023-12-04T05:39:36.886 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.881+0000 7f7c13646640 1 -- 172.21.15.195:0/152400351 shutdown_connections 2023-12-04T05:39:36.887 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.881+0000 7f7c13646640 1 --2- 172.21.15.195:0/152400351 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7c0c15d450 0x7f7c0c1e59e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.887 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.881+0000 7f7c13646640 1 --2- 172.21.15.195:0/152400351 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7c0c153e00 0x7f7c0c070790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.887 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.881+0000 7f7c13646640 1 --2- 172.21.15.195:0/152400351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c0c066450 0x7f7c0c1f3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:36.887 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.881+0000 7f7c13646640 1 -- 172.21.15.195:0/152400351 >> 172.21.15.195:0/152400351 conn(0x7f7c0c06ada0 msgr2=0x7f7c0c0c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:36.887 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.881+0000 7f7c13646640 1 -- 172.21.15.195:0/152400351 shutdown_connections 2023-12-04T05:39:36.887 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:36.881+0000 7f7c13646640 1 -- 172.21.15.195:0/152400351 wait complete. 2023-12-04T05:39:36.891 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:36.889+0000 7f7cb9017180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-04T05:39:36.900 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new db840d83-5a7d-468a-8aa2-1997945847c5 5 2023-12-04T05:39:36.937 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:36.933+0000 7fdb3fdd0180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-04T05:39:36.946 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:36.941+0000 7f7cb9017180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-04T05:39:36.997 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:36.993+0000 7fdb3fdd0180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-04T05:39:37.012 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.009+0000 7f3130b0c640 1 Processor -- start 2023-12-04T05:39:37.012 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.009+0000 7f3130b0c640 1 -- start start 2023-12-04T05:39:37.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.009+0000 7f3130b0c640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f312c153fa0 0x7f312c06a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.009+0000 7f3130b0c640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f312c15dcd0 0x7f312c06aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.009+0000 7f3130b0c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f312c070d40 con 0x7f312c06b240 2023-12-04T05:39:37.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.009+0000 7f3130b0c640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f312c070f70 con 0x7f312c153fa0 2023-12-04T05:39:37.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.009+0000 7f3130b0c640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312c15a0c0 con 0x7f312c15dcd0 2023-12-04T05:39:37.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.009+0000 7f312ad9d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f312c15dcd0 0x7f312c06aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.009+0000 7f312a59c640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f312c153fa0 0x7f312c06a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.014 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.009+0000 7f312ad9d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f312c15dcd0 0x7f312c06aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.195:39620/0 (socket says 172.21.15.195:39620) 2023-12-04T05:39:37.014 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.009+0000 7f312ad9d640 1 -- 172.21.15.195:0/2341778547 learned_addr learned my addr 172.21.15.195:0/2341778547 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-12-04T05:39:37.014 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.009+0000 7f312ad9d640 1 -- 172.21.15.195:0/2341778547 >> v2:172.21.15.195:3300/0 conn(0x7f312c153fa0 msgr2=0x7f312c06a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:37.014 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.009+0000 7f312ad9d640 1 --2- 172.21.15.195:0/2341778547 >> v2:172.21.15.195:3300/0 conn(0x7f312c153fa0 0x7f312c06a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.014 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.009+0000 7f312ad9d640 1 -- 172.21.15.195:0/2341778547 >> v1:172.21.15.107:6789/0 conn(0x7f312c06b240 legacy=0x7f312c15d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:37.014 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.009+0000 7f312ad9d640 1 -- 172.21.15.195:0/2341778547 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312c15a3a0 con 0x7f312c15dcd0 2023-12-04T05:39:37.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.013+0000 7f312ad9d640 1 --2- 172.21.15.195:0/2341778547 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f312c15dcd0 0x7f312c06aaf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3124009e50 tx=0x7f312401f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a947798f81e04ccf server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:37.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.013+0000 7f3129d9b640 1 -- 172.21.15.195:0/2341778547 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31240af400 con 0x7f312c15dcd0 2023-12-04T05:39:37.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.013+0000 7f3129d9b640 1 -- 172.21.15.195:0/2341778547 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31240ab050 con 0x7f312c15dcd0 2023-12-04T05:39:37.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.013+0000 7f3129d9b640 1 -- 172.21.15.195:0/2341778547 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31240af780 con 0x7f312c15dcd0 2023-12-04T05:39:37.016 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.013+0000 7f3130b0c640 1 -- 172.21.15.195:0/2341778547 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f312c15dcd0 msgr2=0x7f312c06aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:37.016 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.013+0000 7f3130b0c640 1 --2- 172.21.15.195:0/2341778547 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f312c15dcd0 0x7f312c06aaf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3124009e50 tx=0x7f312401f1f0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.016 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.013+0000 7f3130b0c640 1 -- 172.21.15.195:0/2341778547 shutdown_connections 2023-12-04T05:39:37.016 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.013+0000 7f3130b0c640 1 --2- 172.21.15.195:0/2341778547 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f312c15dcd0 0x7f312c06aaf0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.016 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.013+0000 7f3130b0c640 1 --2- 172.21.15.195:0/2341778547 >> v2:172.21.15.195:3300/0 conn(0x7f312c153fa0 0x7f312c06a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.017 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.013+0000 7f3130b0c640 1 -- 172.21.15.195:0/2341778547 >> 172.21.15.195:0/2341778547 conn(0x7f312c06b6c0 msgr2=0x7f312c06bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:37.017 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.013+0000 7f3130b0c640 1 -- 172.21.15.195:0/2341778547 shutdown_connections 2023-12-04T05:39:37.017 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.013+0000 7f3130b0c640 1 -- 172.21.15.195:0/2341778547 wait complete. 2023-12-04T05:39:37.017 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.013+0000 7f3130b0c640 1 Processor -- start 2023-12-04T05:39:37.017 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.013+0000 7f3130b0c640 1 -- start start 2023-12-04T05:39:37.019 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f3130b0c640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f312c06b240 0x7f312c0c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.019 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f3130b0c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f312c153fa0 0x7f312c15beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.019 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f3130b0c640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f312c15dcd0 0x7f312c1e31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.020 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f3130b0c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312c1f3270 con 0x7f312c153fa0 2023-12-04T05:39:37.020 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f3130b0c640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312c1f2f90 con 0x7f312c06b240 2023-12-04T05:39:37.020 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f3130b0c640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312c15b0f0 con 0x7f312c15dcd0 2023-12-04T05:39:37.020 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f312a59c640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f312c06b240 0x7f312c0c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.020 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f312a59c640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f312c06b240 0x7f312c0c7b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:37412/0 (socket says 172.21.15.195:37412) 2023-12-04T05:39:37.020 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f312a59c640 1 -- 172.21.15.195:0/389278507 learned_addr learned my addr 172.21.15.195:0/389278507 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-12-04T05:39:37.020 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f312ad9d640 1 --2- 172.21.15.195:0/389278507 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f312c15dcd0 0x7f312c1e31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.020 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f3121d9b640 1 --2- 172.21.15.195:0/389278507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f312c153fa0 0x7f312c15beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.020 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f312a59c640 1 -- 172.21.15.195:0/389278507 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f312c15dcd0 msgr2=0x7f312c1e31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:37.020 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f312a59c640 1 --2- 172.21.15.195:0/389278507 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f312c15dcd0 0x7f312c1e31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.020 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f312a59c640 1 -- 172.21.15.195:0/389278507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f312c153fa0 msgr2=0x7f312c15beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:37.020 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f312a59c640 1 --2- 172.21.15.195:0/389278507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f312c153fa0 0x7f312c15beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.020 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f312a59c640 1 -- 172.21.15.195:0/389278507 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3124009b30 con 0x7f312c06b240 2023-12-04T05:39:37.020 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f3121d9b640 1 --2- 172.21.15.195:0/389278507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f312c153fa0 0x7f312c15beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:39:37.021 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f312a59c640 1 --2- 172.21.15.195:0/389278507 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f312c06b240 0x7f312c0c7b90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f311c00e930 tx=0x7f311c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:37.021 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f31237fe640 1 -- 172.21.15.195:0/389278507 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c009690 con 0x7f312c06b240 2023-12-04T05:39:37.021 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f31237fe640 1 -- 172.21.15.195:0/389278507 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f311c00ca00 con 0x7f312c06b240 2023-12-04T05:39:37.021 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f31237fe640 1 -- 172.21.15.195:0/389278507 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c00cd70 con 0x7f312c06b240 2023-12-04T05:39:37.021 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f3130b0c640 1 -- 172.21.15.195:0/389278507 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f312c15b3d0 con 0x7f312c06b240 2023-12-04T05:39:37.021 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f3130b0c640 1 -- 172.21.15.195:0/389278507 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f312c15b990 con 0x7f312c06b240 2023-12-04T05:39:37.021 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f31237fe640 1 -- 172.21.15.195:0/389278507 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f311c013770 con 0x7f312c06b240 2023-12-04T05:39:37.021 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f31237fe640 1 -- 172.21.15.195:0/389278507 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f311c0123d0 con 0x7f312c06b240 2023-12-04T05:39:37.021 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.017+0000 7f3130b0c640 1 -- 172.21.15.195:0/389278507 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f312c159d40 con 0x7f312c06b240 2023-12-04T05:39:37.024 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.021+0000 7f31237fe640 1 -- 172.21.15.195:0/389278507 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f311c021060 con 0x7f312c06b240 2023-12-04T05:39:37.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.085+0000 7f3130b0c640 1 -- 172.21.15.195:0/389278507 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd new", "uuid": "db840d83-5a7d-468a-8aa2-1997945847c5", "id": 5} v 0) v1 -- 0x7f312c067ec0 con 0x7f312c06b240 2023-12-04T05:39:37.092 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:37.089+0000 7f7cb9017180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-04T05:39:37.095 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.093+0000 7f31237fe640 1 -- 172.21.15.195:0/389278507 <== mon.1 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "db840d83-5a7d-468a-8aa2-1997945847c5", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f311c02a0b0 con 0x7f312c06b240 2023-12-04T05:39:37.095 INFO:teuthology.orchestra.run.smithi195.stdout:5 2023-12-04T05:39:37.096 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.093+0000 7f3130b0c640 1 -- 172.21.15.195:0/389278507 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f312c06b240 msgr2=0x7f312c0c7b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:37.096 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.093+0000 7f3130b0c640 1 --2- 172.21.15.195:0/389278507 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f312c06b240 0x7f312c0c7b90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f311c00e930 tx=0x7f311c00edf0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.096 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.093+0000 7f3130b0c640 1 -- 172.21.15.195:0/389278507 shutdown_connections 2023-12-04T05:39:37.096 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.093+0000 7f3130b0c640 1 --2- 172.21.15.195:0/389278507 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f312c15dcd0 0x7f312c1e31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.096 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.093+0000 7f3130b0c640 1 --2- 172.21.15.195:0/389278507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f312c153fa0 0x7f312c15beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.097 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.093+0000 7f3130b0c640 1 --2- 172.21.15.195:0/389278507 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f312c06b240 0x7f312c0c7b90 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.097 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.093+0000 7f3130b0c640 1 -- 172.21.15.195:0/389278507 >> 172.21.15.195:0/389278507 conn(0x7f312c06b6c0 msgr2=0x7f312c0cab60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:37.097 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.093+0000 7f3130b0c640 1 -- 172.21.15.195:0/389278507 shutdown_connections 2023-12-04T05:39:37.097 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.093+0000 7f3130b0c640 1 -- 172.21.15.195:0/389278507 wait complete. 2023-12-04T05:39:37.111 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 38b5f165-bf91-467e-a615-4b4784337689 6 2023-12-04T05:39:37.154 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:37.149+0000 7f7cb9017180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-04T05:39:37.268 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.265+0000 7f63c051a640 1 Processor -- start 2023-12-04T05:39:37.268 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.265+0000 7f63c051a640 1 -- start start 2023-12-04T05:39:37.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.265+0000 7f63c051a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f63b815d700 0x7f63b815dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.265+0000 7f63c051a640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f63b806b3a0 0x7f63b80c5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.265+0000 7f63c051a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b815e020 con 0x7f63b80c6380 2023-12-04T05:39:37.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.265+0000 7f63c051a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63b8071130 con 0x7f63b815d700 2023-12-04T05:39:37.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.265+0000 7f63c051a640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63b8071410 con 0x7f63b806b3a0 2023-12-04T05:39:37.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.265+0000 7f63beab7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f63b806b3a0 0x7f63b80c5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.265+0000 7f63beab7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f63b806b3a0 0x7f63b80c5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:37418/0 (socket says 172.21.15.195:37418) 2023-12-04T05:39:37.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.265+0000 7f63beab7640 1 -- 172.21.15.195:0/4159226083 learned_addr learned my addr 172.21.15.195:0/4159226083 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-12-04T05:39:37.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.269+0000 7f63be2b6640 1 --2- 172.21.15.195:0/4159226083 >> v2:172.21.15.107:3300/0 conn(0x7f63b815d700 0x7f63b815dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.269+0000 7f63beab7640 1 -- 172.21.15.195:0/4159226083 >> v2:172.21.15.107:3300/0 conn(0x7f63b815d700 msgr2=0x7f63b815dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:37.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.269+0000 7f63beab7640 1 --2- 172.21.15.195:0/4159226083 >> v2:172.21.15.107:3300/0 conn(0x7f63b815d700 0x7f63b815dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.269+0000 7f63beab7640 1 -- 172.21.15.195:0/4159226083 >> v1:172.21.15.107:6789/0 conn(0x7f63b80c6380 legacy=0x7f63b806ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:37.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.269+0000 7f63beab7640 1 -- 172.21.15.195:0/4159226083 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b80716f0 con 0x7f63b806b3a0 2023-12-04T05:39:37.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.269+0000 7f63be2b6640 1 --2- 172.21.15.195:0/4159226083 >> v2:172.21.15.107:3300/0 conn(0x7f63b815d700 0x7f63b815dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:39:37.272 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.269+0000 7f63beab7640 1 --2- 172.21.15.195:0/4159226083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63b806b3a0 0x7f63b80c5260 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f63b80c8500 tx=0x7f63b001ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50671fb0030791f2 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:37.272 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.269+0000 7f63bd2b4640 1 -- 172.21.15.195:0/4159226083 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b00a89b0 con 0x7f63b806b3a0 2023-12-04T05:39:37.272 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.269+0000 7f63bd2b4640 1 -- 172.21.15.195:0/4159226083 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b00a8b10 con 0x7f63b806b3a0 2023-12-04T05:39:37.272 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.269+0000 7f63bd2b4640 1 -- 172.21.15.195:0/4159226083 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b00a8ea0 con 0x7f63b806b3a0 2023-12-04T05:39:37.273 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.269+0000 7f63c051a640 1 -- 172.21.15.195:0/4159226083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63b806b3a0 msgr2=0x7f63b80c5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:37.273 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.269+0000 7f63c051a640 1 --2- 172.21.15.195:0/4159226083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63b806b3a0 0x7f63b80c5260 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f63b80c8500 tx=0x7f63b001ecb0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.273 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.269+0000 7f63c051a640 1 -- 172.21.15.195:0/4159226083 shutdown_connections 2023-12-04T05:39:37.273 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.269+0000 7f63c051a640 1 --2- 172.21.15.195:0/4159226083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63b806b3a0 0x7f63b80c5260 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.273 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.269+0000 7f63c051a640 1 --2- 172.21.15.195:0/4159226083 >> v2:172.21.15.107:3300/0 conn(0x7f63b815d700 0x7f63b815dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.273 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.269+0000 7f63c051a640 1 -- 172.21.15.195:0/4159226083 >> 172.21.15.195:0/4159226083 conn(0x7f63b806e6f0 msgr2=0x7f63b80c8580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:37.274 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.269+0000 7f63c051a640 1 -- 172.21.15.195:0/4159226083 shutdown_connections 2023-12-04T05:39:37.274 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.269+0000 7f63c051a640 1 -- 172.21.15.195:0/4159226083 wait complete. 2023-12-04T05:39:37.276 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63c051a640 1 Processor -- start 2023-12-04T05:39:37.276 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63c051a640 1 -- start start 2023-12-04T05:39:37.276 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63c051a640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63b806b3a0 0x7f63b81efd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.276 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63c051a640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63b80c6380 0x7f63b80bfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.276 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63c051a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63b815d700 0x7f63b80c05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.276 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63c051a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b815c000 con 0x7f63b815d700 2023-12-04T05:39:37.276 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63c051a640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b80beea0 con 0x7f63b806b3a0 2023-12-04T05:39:37.276 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63c051a640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b80c0ff0 con 0x7f63b80c6380 2023-12-04T05:39:37.276 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63be2b6640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63b806b3a0 0x7f63b81efd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.276 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63bdab5640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63b80c6380 0x7f63b80bfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.276 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63be2b6640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63b806b3a0 0x7f63b81efd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:37430/0 (socket says 172.21.15.195:37430) 2023-12-04T05:39:37.276 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63be2b6640 1 -- 172.21.15.195:0/414724587 learned_addr learned my addr 172.21.15.195:0/414724587 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-12-04T05:39:37.276 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63beab7640 1 --2- 172.21.15.195:0/414724587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63b815d700 0x7f63b80c05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.277 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63be2b6640 1 -- 172.21.15.195:0/414724587 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63b80c6380 msgr2=0x7f63b80bfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:37.277 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63be2b6640 1 --2- 172.21.15.195:0/414724587 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63b80c6380 0x7f63b80bfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.277 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63be2b6640 1 -- 172.21.15.195:0/414724587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63b815d700 msgr2=0x7f63b80c05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:37.277 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63be2b6640 1 --2- 172.21.15.195:0/414724587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63b815d700 0x7f63b80c05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.277 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63be2b6640 1 -- 172.21.15.195:0/414724587 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b00094d0 con 0x7f63b806b3a0 2023-12-04T05:39:37.277 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63bdab5640 1 --2- 172.21.15.195:0/414724587 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63b80c6380 0x7f63b80bfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T05:39:37.277 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63be2b6640 1 --2- 172.21.15.195:0/414724587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63b806b3a0 0x7f63b81efd70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f63b400ce00 tx=0x7f63b400a650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:37.277 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63aeffd640 1 -- 172.21.15.195:0/414724587 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b400f890 con 0x7f63b806b3a0 2023-12-04T05:39:37.277 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63aeffd640 1 -- 172.21.15.195:0/414724587 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b400f9f0 con 0x7f63b806b3a0 2023-12-04T05:39:37.277 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63aeffd640 1 -- 172.21.15.195:0/414724587 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b400fdc0 con 0x7f63b806b3a0 2023-12-04T05:39:37.277 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63c051a640 1 -- 172.21.15.195:0/414724587 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b80c12d0 con 0x7f63b806b3a0 2023-12-04T05:39:37.277 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63c051a640 1 -- 172.21.15.195:0/414724587 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b815b9b0 con 0x7f63b806b3a0 2023-12-04T05:39:37.277 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63aeffd640 1 -- 172.21.15.195:0/414724587 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f63b400b040 con 0x7f63b806b3a0 2023-12-04T05:39:37.277 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63aeffd640 1 -- 172.21.15.195:0/414724587 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f63b400ae20 con 0x7f63b806b3a0 2023-12-04T05:39:37.278 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.273+0000 7f63c051a640 1 -- 172.21.15.195:0/414724587 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b8067ec0 con 0x7f63b806b3a0 2023-12-04T05:39:37.279 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.277+0000 7f63aeffd640 1 -- 172.21.15.195:0/414724587 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f63b401d020 con 0x7f63b806b3a0 2023-12-04T05:39:37.342 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.341+0000 7f63c051a640 1 -- 172.21.15.195:0/414724587 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd new", "uuid": "38b5f165-bf91-467e-a615-4b4784337689", "id": 6} v 0) v1 -- 0x7f63b8066590 con 0x7f63b806b3a0 2023-12-04T05:39:37.349 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.345+0000 7f63aeffd640 1 -- 172.21.15.195:0/414724587 <== mon.1 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "38b5f165-bf91-467e-a615-4b4784337689", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f63b4018700 con 0x7f63b806b3a0 2023-12-04T05:39:37.349 INFO:teuthology.orchestra.run.smithi195.stdout:6 2023-12-04T05:39:37.350 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.349+0000 7f63c051a640 1 -- 172.21.15.195:0/414724587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63b806b3a0 msgr2=0x7f63b81efd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:37.350 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.349+0000 7f63c051a640 1 --2- 172.21.15.195:0/414724587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63b806b3a0 0x7f63b81efd70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f63b400ce00 tx=0x7f63b400a650 comp rx=0 tx=0).stop 2023-12-04T05:39:37.350 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.349+0000 7f63c051a640 1 -- 172.21.15.195:0/414724587 shutdown_connections 2023-12-04T05:39:37.351 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.349+0000 7f63c051a640 1 --2- 172.21.15.195:0/414724587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63b815d700 0x7f63b80c05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.351 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.349+0000 7f63c051a640 1 --2- 172.21.15.195:0/414724587 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f63b80c6380 0x7f63b80bfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.351 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.349+0000 7f63c051a640 1 --2- 172.21.15.195:0/414724587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63b806b3a0 0x7f63b81efd70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.351 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.349+0000 7f63c051a640 1 -- 172.21.15.195:0/414724587 >> 172.21.15.195:0/414724587 conn(0x7f63b806e6f0 msgr2=0x7f63b80c5030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:37.351 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.349+0000 7f63c051a640 1 -- 172.21.15.195:0/414724587 shutdown_connections 2023-12-04T05:39:37.351 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.349+0000 7f63c051a640 1 -- 172.21.15.195:0/414724587 wait complete. 2023-12-04T05:39:37.365 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new e5b3c596-494b-4df3-9437-bc2d52168ac4 7 2023-12-04T05:39:37.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.481+0000 7f068fefd640 1 Processor -- start 2023-12-04T05:39:37.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.481+0000 7f068fefd640 1 -- start start 2023-12-04T05:39:37.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.485+0000 7f068fefd640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f06880c8c10 0x7f06881581d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.485+0000 7f068fefd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0688158710 0x7f06880c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.485+0000 7f068fefd640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0688158b60 con 0x7f068815e6b0 2023-12-04T05:39:37.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.485+0000 7f068fefd640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06880c6290 con 0x7f0688158710 2023-12-04T05:39:37.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.485+0000 7f068fefd640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06880c5ef0 con 0x7f06880c8c10 2023-12-04T05:39:37.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.485+0000 7f068dc99640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f068815e6b0 0x7f068815ea90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:55270/0 (socket says 172.21.15.195:55270) 2023-12-04T05:39:37.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.485+0000 7f068dc99640 1 -- 172.21.15.195:0/2272819836 learned_addr learned my addr 172.21.15.195:0/2272819836 (peer_addr_for_me v1:172.21.15.195:0/0) 2023-12-04T05:39:37.489 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.485+0000 7f068e49a640 1 --2- 172.21.15.195:0/2272819836 >> v2:172.21.15.107:3300/0 conn(0x7f0688158710 0x7f06880c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.489 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.485+0000 7f068d498640 1 --2- 172.21.15.195:0/2272819836 >> v2:172.21.15.195:3300/0 conn(0x7f06880c8c10 0x7f06881581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.489 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.485+0000 7f068cc97640 1 -- 172.21.15.195:0/2272819836 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302265285 0 0) 0x7f0688158b60 con 0x7f068815e6b0 2023-12-04T05:39:37.489 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.485+0000 7f068cc97640 1 -- 172.21.15.195:0/2272819836 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0674003530 con 0x7f068815e6b0 2023-12-04T05:39:37.489 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.485+0000 7f068e49a640 1 -- 172.21.15.195:0/2272819836 >> v2:172.21.15.195:3300/0 conn(0x7f06880c8c10 msgr2=0x7f06881581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:37.489 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.485+0000 7f068cc97640 1 -- 172.21.15.195:0/2272819836 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1575617574 0 0) 0x7f0674003530 con 0x7f068815e6b0 2023-12-04T05:39:37.489 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.485+0000 7f068e49a640 1 --2- 172.21.15.195:0/2272819836 >> v2:172.21.15.195:3300/0 conn(0x7f06880c8c10 0x7f06881581d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.490 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.485+0000 7f068e49a640 1 -- 172.21.15.195:0/2272819836 >> v1:172.21.15.195:6789/0 conn(0x7f068815e6b0 legacy=0x7f068815ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:37.490 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.485+0000 7f068e49a640 1 -- 172.21.15.195:0/2272819836 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06880c25e0 con 0x7f0688158710 2023-12-04T05:39:37.490 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.485+0000 7f068d498640 1 --2- 172.21.15.195:0/2272819836 >> v2:172.21.15.195:3300/0 conn(0x7f06880c8c10 0x7f06881581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T05:39:37.492 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.489+0000 7f068e49a640 1 --2- 172.21.15.195:0/2272819836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0688158710 0x7f06880c66b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0688074700 tx=0x7f068401eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd33577bb69f313b server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:37.492 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.489+0000 7f068cc97640 1 -- 172.21.15.195:0/2272819836 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0680001a00 con 0x7f0688158710 2023-12-04T05:39:37.492 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.489+0000 7f068cc97640 1 -- 172.21.15.195:0/2272819836 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06840a85f0 con 0x7f0688158710 2023-12-04T05:39:37.493 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.489+0000 7f068fefd640 1 -- 172.21.15.195:0/2272819836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0688158710 msgr2=0x7f06880c66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:37.494 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.489+0000 7f068fefd640 1 --2- 172.21.15.195:0/2272819836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0688158710 0x7f06880c66b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0688074700 tx=0x7f068401eb70 comp rx=0 tx=0).stop 2023-12-04T05:39:37.494 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.489+0000 7f068fefd640 1 -- 172.21.15.195:0/2272819836 shutdown_connections 2023-12-04T05:39:37.494 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.489+0000 7f068fefd640 1 --2- 172.21.15.195:0/2272819836 >> v2:172.21.15.195:3300/0 conn(0x7f06880c8c10 0x7f06881581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.494 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.489+0000 7f068fefd640 1 --2- 172.21.15.195:0/2272819836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0688158710 0x7f06880c66b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.494 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.489+0000 7f068fefd640 1 -- 172.21.15.195:0/2272819836 >> 172.21.15.195:0/2272819836 conn(0x7f06880c7ff0 msgr2=0x7f06880c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:37.494 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.489+0000 7f068fefd640 1 -- 172.21.15.195:0/2272819836 shutdown_connections 2023-12-04T05:39:37.494 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.489+0000 7f068fefd640 1 -- 172.21.15.195:0/2272819836 wait complete. 2023-12-04T05:39:37.495 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.493+0000 7f068fefd640 1 Processor -- start 2023-12-04T05:39:37.496 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.493+0000 7f068fefd640 1 -- start start 2023-12-04T05:39:37.496 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.493+0000 7f068fefd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06880c8c10 0x7f06880c3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.496 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.493+0000 7f068fefd640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0688158710 0x7f06880c55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.496 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.493+0000 7f068fefd640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f068815e6b0 0x7f068815d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.496 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.493+0000 7f068fefd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06880cb2c0 con 0x7f06880c8c10 2023-12-04T05:39:37.496 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.493+0000 7f068fefd640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068815dbf0 con 0x7f068815e6b0 2023-12-04T05:39:37.496 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.493+0000 7f068fefd640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f068815c330 con 0x7f0688158710 2023-12-04T05:39:37.496 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.493+0000 7f068e49a640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f068815e6b0 0x7f068815d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.496 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.493+0000 7f068e49a640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f068815e6b0 0x7f068815d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:37446/0 (socket says 172.21.15.195:37446) 2023-12-04T05:39:37.496 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.493+0000 7f068e49a640 1 -- 172.21.15.195:0/344422103 learned_addr learned my addr 172.21.15.195:0/344422103 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-12-04T05:39:37.496 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.493+0000 7f068dc99640 1 --2- 172.21.15.195:0/344422103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06880c8c10 0x7f06880c3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.496 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.493+0000 7f068d498640 1 --2- 172.21.15.195:0/344422103 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0688158710 0x7f06880c55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.497 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.493+0000 7f068e49a640 1 -- 172.21.15.195:0/344422103 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0688158710 msgr2=0x7f06880c55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:37.497 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.493+0000 7f068e49a640 1 --2- 172.21.15.195:0/344422103 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0688158710 0x7f06880c55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.497 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.493+0000 7f068e49a640 1 -- 172.21.15.195:0/344422103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06880c8c10 msgr2=0x7f06880c3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:37.497 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.493+0000 7f068e49a640 1 --2- 172.21.15.195:0/344422103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06880c8c10 0x7f06880c3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.497 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.493+0000 7f068e49a640 1 -- 172.21.15.195:0/344422103 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0678009420 con 0x7f068815e6b0 2023-12-04T05:39:37.497 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.493+0000 7f068d498640 1 --2- 172.21.15.195:0/344422103 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0688158710 0x7f06880c55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:39:37.497 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.493+0000 7f068e49a640 1 --2- 172.21.15.195:0/344422103 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f068815e6b0 0x7f068815d6b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f06840b3040 tx=0x7f068401b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:37.499 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.497+0000 7f067e7fc640 1 -- 172.21.15.195:0/344422103 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06840b0a50 con 0x7f068815e6b0 2023-12-04T05:39:37.499 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.497+0000 7f067e7fc640 1 -- 172.21.15.195:0/344422103 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06840b0bb0 con 0x7f068815e6b0 2023-12-04T05:39:37.499 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.497+0000 7f068fefd640 1 -- 172.21.15.195:0/344422103 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0684009420 con 0x7f068815e6b0 2023-12-04T05:39:37.499 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.497+0000 7f067e7fc640 1 -- 172.21.15.195:0/344422103 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06840b9410 con 0x7f068815e6b0 2023-12-04T05:39:37.499 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.497+0000 7f068fefd640 1 -- 172.21.15.195:0/344422103 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f068815c8f0 con 0x7f068815e6b0 2023-12-04T05:39:37.500 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.497+0000 7f068fefd640 1 -- 172.21.15.195:0/344422103 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0688067870 con 0x7f068815e6b0 2023-12-04T05:39:37.500 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.497+0000 7f067e7fc640 1 -- 172.21.15.195:0/344422103 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f06840b7040 con 0x7f068815e6b0 2023-12-04T05:39:37.501 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.497+0000 7f067e7fc640 1 -- 172.21.15.195:0/344422103 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f06840ce3d0 con 0x7f068815e6b0 2023-12-04T05:39:37.505 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.501+0000 7f067e7fc640 1 -- 172.21.15.195:0/344422103 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f06840b5070 con 0x7f068815e6b0 2023-12-04T05:39:37.585 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.581+0000 7f068fefd640 1 -- 172.21.15.195:0/344422103 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e5b3c596-494b-4df3-9437-bc2d52168ac4", "id": 7} v 0) v1 -- 0x7f06880c6f30 con 0x7f068815e6b0 2023-12-04T05:39:37.592 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.589+0000 7f067e7fc640 1 -- 172.21.15.195:0/344422103 <== mon.1 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e5b3c596-494b-4df3-9437-bc2d52168ac4", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f06840b9570 con 0x7f068815e6b0 2023-12-04T05:39:37.592 INFO:teuthology.orchestra.run.smithi195.stdout:7 2023-12-04T05:39:37.593 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.589+0000 7f068fefd640 1 -- 172.21.15.195:0/344422103 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f068815e6b0 msgr2=0x7f068815d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:37.593 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.589+0000 7f068fefd640 1 --2- 172.21.15.195:0/344422103 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f068815e6b0 0x7f068815d6b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f06840b3040 tx=0x7f068401b910 comp rx=0 tx=0).stop 2023-12-04T05:39:37.593 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.589+0000 7f068fefd640 1 -- 172.21.15.195:0/344422103 shutdown_connections 2023-12-04T05:39:37.593 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.589+0000 7f068fefd640 1 --2- 172.21.15.195:0/344422103 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f068815e6b0 0x7f068815d6b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.593 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.589+0000 7f068fefd640 1 --2- 172.21.15.195:0/344422103 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0688158710 0x7f06880c55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.593 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.589+0000 7f068fefd640 1 --2- 172.21.15.195:0/344422103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06880c8c10 0x7f06880c3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.593 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.589+0000 7f068fefd640 1 -- 172.21.15.195:0/344422103 >> 172.21.15.195:0/344422103 conn(0x7f06880c7ff0 msgr2=0x7f06881566e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:37.594 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.589+0000 7f068fefd640 1 -- 172.21.15.195:0/344422103 shutdown_connections 2023-12-04T05:39:37.594 INFO:teuthology.orchestra.run.smithi195.stderr:2023-12-04T05:39:37.589+0000 7f068fefd640 1 -- 172.21.15.195:0/344422103 wait complete. 2023-12-04T05:39:37.607 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-04T05:39:37.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-12-04T05:39:37.610 INFO:tasks.ceph.osd.0:Started 2023-12-04T05:39:37.610 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-04T05:39:37.610 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-12-04T05:39:37.613 INFO:tasks.ceph.osd.1:Started 2023-12-04T05:39:37.613 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-04T05:39:37.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-12-04T05:39:37.615 INFO:tasks.ceph.osd.2:Started 2023-12-04T05:39:37.615 INFO:tasks.ceph.osd.3:Restarting daemon 2023-12-04T05:39:37.615 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-12-04T05:39:37.617 INFO:tasks.ceph.osd.3:Started 2023-12-04T05:39:37.617 INFO:tasks.ceph.osd.4:Restarting daemon 2023-12-04T05:39:37.618 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-12-04T05:39:37.620 INFO:tasks.ceph.osd.4:Started 2023-12-04T05:39:37.620 INFO:tasks.ceph.osd.5:Restarting daemon 2023-12-04T05:39:37.620 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-12-04T05:39:37.622 INFO:tasks.ceph.osd.5:Started 2023-12-04T05:39:37.622 INFO:tasks.ceph.osd.6:Restarting daemon 2023-12-04T05:39:37.622 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-12-04T05:39:37.624 INFO:tasks.ceph.osd.6:Started 2023-12-04T05:39:37.624 INFO:tasks.ceph.osd.7:Restarting daemon 2023-12-04T05:39:37.624 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-12-04T05:39:37.626 INFO:tasks.ceph.osd.7:Started 2023-12-04T05:39:37.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T05:39:37.646 INFO:tasks.ceph.mgr.y.smithi107.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-12-04T05:39:37.646 INFO:tasks.ceph.mgr.y.smithi107.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-12-04T05:39:37.646 INFO:tasks.ceph.mgr.y.smithi107.stderr: from numpy import show_config as show_numpy_config 2023-12-04T05:39:37.706 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:37.701+0000 7fdb3fdd0180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-04T05:39:37.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.753+0000 7f27abdcf640 1 Processor -- start 2023-12-04T05:39:37.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.753+0000 7f27abdcf640 1 -- start start 2023-12-04T05:39:37.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27abdcf640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f27a4071030 0x7f27a40c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27abdcf640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f27a40c60e0 0x7f27a406e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27abdcf640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27a4152360 con 0x7f27a4151f80 2023-12-04T05:39:37.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27abdcf640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27a406acf0 con 0x7f27a40c60e0 2023-12-04T05:39:37.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27abdcf640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a4071430 con 0x7f27a4071030 2023-12-04T05:39:37.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27a9b6b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f27a4151f80 0x7f27a406c2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57696/0 (socket says 172.21.15.107:57696) 2023-12-04T05:39:37.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27a9b6b640 1 -- 172.21.15.107:0/43901389 learned_addr learned my addr 172.21.15.107:0/43901389 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:39:37.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27aa36c640 1 --2- 172.21.15.107:0/43901389 >> v2:172.21.15.195:3300/0 conn(0x7f27a40c60e0 0x7f27a406e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27a936a640 1 --2- 172.21.15.107:0/43901389 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f27a4071030 0x7f27a40c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27a8b69640 1 -- 172.21.15.107:0/43901389 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930512263 0 0) 0x7f27a4152360 con 0x7f27a4151f80 2023-12-04T05:39:37.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27a8b69640 1 -- 172.21.15.107:0/43901389 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2790003530 con 0x7f27a4151f80 2023-12-04T05:39:37.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27a8b69640 1 -- 172.21.15.107:0/43901389 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2705236083 0 0) 0x7f2790003530 con 0x7f27a4151f80 2023-12-04T05:39:37.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27a8b69640 1 -- 172.21.15.107:0/43901389 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f27a4071030 msgr2=0x7f27a40c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:37.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27a8b69640 1 --2- 172.21.15.107:0/43901389 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f27a4071030 0x7f27a40c5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27a8b69640 1 -- 172.21.15.107:0/43901389 >> v2:172.21.15.195:3300/0 conn(0x7f27a40c60e0 msgr2=0x7f27a406e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:37.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27a8b69640 1 --2- 172.21.15.107:0/43901389 >> v2:172.21.15.195:3300/0 conn(0x7f27a40c60e0 0x7f27a406e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27a8b69640 1 -- 172.21.15.107:0/43901389 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a4157c90 con 0x7f27a4151f80 2023-12-04T05:39:37.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27a8b69640 1 -- 172.21.15.107:0/43901389 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f2794001680 con 0x7f27a4151f80 2023-12-04T05:39:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27a8b69640 1 -- 172.21.15.107:0/43901389 >> v1:172.21.15.107:6789/0 conn(0x7f27a4151f80 legacy=0x7f27a406c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27a8b69640 1 --2- 172.21.15.107:0/43901389 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f27a4071030 0x7f2790003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27a8b69640 1 --2- 172.21.15.107:0/43901389 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2790003f00 0x7f27900047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27a8b69640 1 --2- 172.21.15.107:0/43901389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2790004d10 0x7f2790005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27a8b69640 1 -- 172.21.15.107:0/43901389 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a406acf0 con 0x7f2790004d10 2023-12-04T05:39:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27a8b69640 1 -- 172.21.15.107:0/43901389 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a4071430 con 0x7f2790003f00 2023-12-04T05:39:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27a8b69640 1 -- 172.21.15.107:0/43901389 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f279000b9c0 con 0x7f27a4071030 2023-12-04T05:39:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27aa36c640 1 --2- 172.21.15.107:0/43901389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2790004d10 0x7f2790005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27a936a640 1 --2- 172.21.15.107:0/43901389 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2790003f00 0x7f27900047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27a9b6b640 1 --2- 172.21.15.107:0/43901389 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f27a4071030 0x7f2790003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27aa36c640 1 -- 172.21.15.107:0/43901389 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f27a4071030 msgr2=0x7f2790003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27aa36c640 1 --2- 172.21.15.107:0/43901389 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f27a4071030 0x7f2790003980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27aa36c640 1 -- 172.21.15.107:0/43901389 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2790003f00 msgr2=0x7f27900047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27aa36c640 1 --2- 172.21.15.107:0/43901389 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2790003f00 0x7f27900047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27aa36c640 1 -- 172.21.15.107:0/43901389 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2790003530 con 0x7f2790004d10 2023-12-04T05:39:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27a936a640 1 --2- 172.21.15.107:0/43901389 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2790003f00 0x7f27900047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:39:37.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.757+0000 7f27aa36c640 1 --2- 172.21.15.107:0/43901389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2790004d10 0x7f2790005190 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f279c009fc0 tx=0x7f279c01cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff75f29533734b8 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:37.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27a8b69640 1 -- 172.21.15.107:0/43901389 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279c0a5cb0 con 0x7f2790004d10 2023-12-04T05:39:37.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27a8b69640 1 -- 172.21.15.107:0/43901389 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f279c00f040 con 0x7f2790004d10 2023-12-04T05:39:37.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27a8b69640 1 -- 172.21.15.107:0/43901389 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279c0ad680 con 0x7f2790004d10 2023-12-04T05:39:37.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27abdcf640 1 -- 172.21.15.107:0/43901389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2790004d10 msgr2=0x7f2790005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:37.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27abdcf640 1 --2- 172.21.15.107:0/43901389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2790004d10 0x7f2790005190 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f279c009fc0 tx=0x7f279c01cca0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27aa36c640 1 -- 172.21.15.107:0/43901389 reap_dead start 2023-12-04T05:39:37.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27abdcf640 1 -- 172.21.15.107:0/43901389 shutdown_connections 2023-12-04T05:39:37.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27abdcf640 1 -- 172.21.15.107:0/43901389 >> 172.21.15.107:0/43901389 conn(0x7f27a40cc490 msgr2=0x7f27a40cc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:37.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27abdcf640 1 -- 172.21.15.107:0/43901389 shutdown_connections 2023-12-04T05:39:37.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27abdcf640 1 -- 172.21.15.107:0/43901389 wait complete. 2023-12-04T05:39:37.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27abdcf640 1 Processor -- start 2023-12-04T05:39:37.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27abdcf640 1 -- start start 2023-12-04T05:39:37.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27abdcf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27a406e240 0x7f27a414c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27abdcf640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f27a4155340 0x7f27a41497f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27abdcf640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f27a406e620 0x7f27a4149d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:37.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27abdcf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a41f1150 con 0x7f27a406e240 2023-12-04T05:39:37.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27abdcf640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a414cc50 con 0x7f27a4155340 2023-12-04T05:39:37.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27abdcf640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a41f0540 con 0x7f27a406e620 2023-12-04T05:39:37.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27a9b6b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27a406e240 0x7f27a414c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27a9b6b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27a406e240 0x7f27a414c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44454/0 (socket says 172.21.15.107:44454) 2023-12-04T05:39:37.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27a9b6b640 1 -- 172.21.15.107:0/1292097912 learned_addr learned my addr 172.21.15.107:0/1292097912 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:39:37.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27aa36c640 1 --2- 172.21.15.107:0/1292097912 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f27a406e620 0x7f27a4149d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27a936a640 1 --2- 172.21.15.107:0/1292097912 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f27a4155340 0x7f27a41497f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:37.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27a9b6b640 1 -- 172.21.15.107:0/1292097912 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f27a406e620 msgr2=0x7f27a4149d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:37.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27a9b6b640 1 --2- 172.21.15.107:0/1292097912 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f27a406e620 0x7f27a4149d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27a9b6b640 1 -- 172.21.15.107:0/1292097912 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f27a4155340 msgr2=0x7f27a41497f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:37.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27a9b6b640 1 --2- 172.21.15.107:0/1292097912 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f27a4155340 0x7f27a41497f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27a9b6b640 1 -- 172.21.15.107:0/1292097912 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a41f0820 con 0x7f27a406e240 2023-12-04T05:39:37.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27a9b6b640 1 --2- 172.21.15.107:0/1292097912 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27a406e240 0x7f27a414c5a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f279400d8b0 tx=0x7f279400dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:37.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f279a7fc640 1 -- 172.21.15.107:0/1292097912 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2794004cc0 con 0x7f27a406e240 2023-12-04T05:39:37.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f279a7fc640 1 -- 172.21.15.107:0/1292097912 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2794004e20 con 0x7f27a406e240 2023-12-04T05:39:37.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f279a7fc640 1 -- 172.21.15.107:0/1292097912 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2794007310 con 0x7f27a406e240 2023-12-04T05:39:37.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27abdcf640 1 -- 172.21.15.107:0/1292097912 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27a41f0b00 con 0x7f27a406e240 2023-12-04T05:39:37.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.761+0000 7f27abdcf640 1 -- 172.21.15.107:0/1292097912 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a4148100 con 0x7f27a406e240 2023-12-04T05:39:37.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.765+0000 7f279a7fc640 1 -- 172.21.15.107:0/1292097912 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2794007470 con 0x7f27a406e240 2023-12-04T05:39:37.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.765+0000 7f279a7fc640 1 -- 172.21.15.107:0/1292097912 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f2794013b40 con 0x7f27a406e240 2023-12-04T05:39:37.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.765+0000 7f27abdcf640 1 -- 172.21.15.107:0/1292097912 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27a4067ec0 con 0x7f27a406e240 2023-12-04T05:39:37.783 INFO:tasks.ceph.mgr.x.smithi195.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-12-04T05:39:37.783 INFO:tasks.ceph.mgr.x.smithi195.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-12-04T05:39:37.783 INFO:tasks.ceph.mgr.x.smithi195.stderr: from numpy import show_config as show_numpy_config 2023-12-04T05:39:37.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.785+0000 7f279a7fc640 1 -- 172.21.15.107:0/1292097912 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f2794019070 con 0x7f27a406e240 2023-12-04T05:39:37.835 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:37.833+0000 7f7cb9017180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-04T05:39:37.838 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:37.833+0000 7fdb3fdd0180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-04T05:39:37.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.841+0000 7f27abdcf640 1 -- 172.21.15.107:0/1292097912 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27a4071940 con 0x7f27a406e240 2023-12-04T05:39:37.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.841+0000 7f279a7fc640 1 -- 172.21.15.107:0/1292097912 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f279400b6b0 con 0x7f27a406e240 2023-12-04T05:39:37.845 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:37.845 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":10,"fsid":"603e856b-31ac-475a-8f35-21ab7faadd48","created":"2023-12-04T05:39:34.921690+0000","modified":"2023-12-04T05:39:37.587639+0000","last_up_change":"0.000000","last_in_change":"2023-12-04T05:39:37.587639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0ab7111c-0ae8-4d41-b6db-e3743614322c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"baef7a1f-8026-4fab-9e70-48465271034e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"90e8cf24-7b54-484a-9d85-b44a9d3dfeab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"54658731-2b9a-4d5e-ab13-7c1be7d34e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9a9dea13-8afe-485d-aa43-45eda691179b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"db840d83-5a7d-468a-8aa2-1997945847c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"38b5f165-bf91-467e-a615-4b4784337689","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e5b3c596-494b-4df3-9437-bc2d52168ac4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T05:39:37.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.841+0000 7f27abdcf640 1 -- 172.21.15.107:0/1292097912 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27a406e240 msgr2=0x7f27a414c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:37.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.841+0000 7f27abdcf640 1 --2- 172.21.15.107:0/1292097912 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27a406e240 0x7f27a414c5a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f279400d8b0 tx=0x7f279400dd70 comp rx=0 tx=0).stop 2023-12-04T05:39:37.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.841+0000 7f27abdcf640 1 -- 172.21.15.107:0/1292097912 shutdown_connections 2023-12-04T05:39:37.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.841+0000 7f27abdcf640 1 --2- 172.21.15.107:0/1292097912 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f27a406e620 0x7f27a4149d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.841+0000 7f27abdcf640 1 --2- 172.21.15.107:0/1292097912 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f27a4155340 0x7f27a41497f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.841+0000 7f27abdcf640 1 --2- 172.21.15.107:0/1292097912 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27a406e240 0x7f27a414c5a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:37.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.841+0000 7f27abdcf640 1 -- 172.21.15.107:0/1292097912 >> 172.21.15.107:0/1292097912 conn(0x7f27a40cc490 msgr2=0x7f27a4071b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:37.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.841+0000 7f27abdcf640 1 -- 172.21.15.107:0/1292097912 shutdown_connections 2023-12-04T05:39:37.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:37.841+0000 7f27abdcf640 1 -- 172.21.15.107:0/1292097912 wait complete. 2023-12-04T05:39:37.863 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-04T05:39:37.863 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-04T05:39:37.977 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:37.973+0000 7f7cb9017180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-04T05:39:38.003 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:38.001+0000 7fdb3fdd0180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-04T05:39:38.066 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:38.061+0000 7fdb3fdd0180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-04T05:39:38.126 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:38.125+0000 7f7cb9017180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-04T05:39:38.188 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:38.185+0000 7fdb3fdd0180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-04T05:39:38.189 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:38.185+0000 7f7cb9017180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-04T05:39:38.224 INFO:tasks.ceph.osd.1.smithi107.stderr:2023-12-04T05:39:38.221+0000 7f4957ccb640 -1 Falling back to public interface 2023-12-04T05:39:38.231 INFO:tasks.ceph.osd.4.smithi195.stderr:2023-12-04T05:39:38.229+0000 7f20f6823640 -1 Falling back to public interface 2023-12-04T05:39:38.235 INFO:tasks.ceph.osd.2.smithi107.stderr:2023-12-04T05:39:38.233+0000 7fad6cc89640 -1 Falling back to public interface 2023-12-04T05:39:38.238 INFO:tasks.ceph.osd.7.smithi195.stderr:2023-12-04T05:39:38.237+0000 7f82fdce9640 -1 Falling back to public interface 2023-12-04T05:39:38.239 INFO:tasks.ceph.osd.6.smithi195.stderr:2023-12-04T05:39:38.237+0000 7fd574750640 -1 Falling back to public interface 2023-12-04T05:39:38.243 INFO:tasks.ceph.osd.0.smithi107.stderr:2023-12-04T05:39:38.241+0000 7fde5bdaf640 -1 Falling back to public interface 2023-12-04T05:39:38.243 INFO:tasks.ceph.osd.3.smithi107.stderr:2023-12-04T05:39:38.241+0000 7f775c774640 -1 Falling back to public interface 2023-12-04T05:39:38.254 INFO:tasks.ceph.osd.5.smithi195.stderr:2023-12-04T05:39:38.253+0000 7f2fc3c87640 -1 Falling back to public interface 2023-12-04T05:39:38.314 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:38.313+0000 7f7cb9017180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-04T05:39:38.670 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-04T05:39:38.782 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.777+0000 7f3273026640 1 Processor -- start 2023-12-04T05:39:38.783 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.777+0000 7f3273026640 1 -- start start 2023-12-04T05:39:38.783 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.781+0000 7f3273026640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f326c0724f0 0x7f326c0c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:38.783 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.781+0000 7f3273026640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f326c06d6e0 con 0x7f326c154160 2023-12-04T05:39:38.784 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.781+0000 7f3273026640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f326c154540 con 0x7f326c15d530 2023-12-04T05:39:38.784 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.781+0000 7f3273026640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f326c15aca0 con 0x7f326c0724f0 2023-12-04T05:39:38.784 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.781+0000 7f326bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f326c0724f0 0x7f326c0c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:38.784 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.781+0000 7f3270dc2640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f326c154160 0x7f326c0cd760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57702/0 (socket says 172.21.15.107:57702) 2023-12-04T05:39:38.784 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.781+0000 7f3270dc2640 1 -- 172.21.15.107:0/2035908127 learned_addr learned my addr 172.21.15.107:0/2035908127 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:39:38.784 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.781+0000 7f326bfff640 1 -- 172.21.15.107:0/2035908127 >> v1:172.21.15.195:6789/0 conn(0x7f326c15d530 legacy=0x7f326c15d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:38.784 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.781+0000 7f326bfff640 1 -- 172.21.15.107:0/2035908127 >> v1:172.21.15.107:6789/0 conn(0x7f326c154160 legacy=0x7f326c0cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:38.784 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.781+0000 7f326b7fe640 1 -- 172.21.15.107:0/2035908127 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252532729 0 0) 0x7f326c06d6e0 con 0x7f326c154160 2023-12-04T05:39:38.785 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.781+0000 7f326bfff640 1 -- 172.21.15.107:0/2035908127 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326c15a020 con 0x7f326c0724f0 2023-12-04T05:39:38.786 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.781+0000 7f326bfff640 1 --2- 172.21.15.107:0/2035908127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f326c0724f0 0x7f326c0c9aa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f32580097c0 tx=0x7f325801ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea60f19e5badb5de server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:38.786 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.781+0000 7f326b7fe640 1 -- 172.21.15.107:0/2035908127 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32580a8930 con 0x7f326c0724f0 2023-12-04T05:39:38.786 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.781+0000 7f326b7fe640 1 -- 172.21.15.107:0/2035908127 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32580a8a90 con 0x7f326c0724f0 2023-12-04T05:39:38.786 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.781+0000 7f326b7fe640 1 -- 172.21.15.107:0/2035908127 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32580a8e20 con 0x7f326c0724f0 2023-12-04T05:39:38.787 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:38.785+0000 7fdb3fdd0180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-04T05:39:38.787 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f3273026640 1 -- 172.21.15.107:0/2035908127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f326c0724f0 msgr2=0x7f326c0c9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:38.788 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f3273026640 1 --2- 172.21.15.107:0/2035908127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f326c0724f0 0x7f326c0c9aa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f32580097c0 tx=0x7f325801ec30 comp rx=0 tx=0).stop 2023-12-04T05:39:38.788 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f3273026640 1 -- 172.21.15.107:0/2035908127 shutdown_connections 2023-12-04T05:39:38.788 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f3273026640 1 --2- 172.21.15.107:0/2035908127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f326c0724f0 0x7f326c0c9aa0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:38.788 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f3273026640 1 -- 172.21.15.107:0/2035908127 >> 172.21.15.107:0/2035908127 conn(0x7f326c06fa00 msgr2=0x7f326c06fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:38.788 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f3273026640 1 -- 172.21.15.107:0/2035908127 shutdown_connections 2023-12-04T05:39:38.788 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f3273026640 1 -- 172.21.15.107:0/2035908127 wait complete. 2023-12-04T05:39:38.790 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f3273026640 1 Processor -- start 2023-12-04T05:39:38.790 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f3273026640 1 -- start start 2023-12-04T05:39:38.790 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f3273026640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f326c0724f0 0x7f326c1efc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:38.791 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f3273026640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f326c154160 0x7f326c153510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:38.791 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f3273026640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f326c15d530 0x7f326c0bdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:38.791 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f3273026640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c0c1870 con 0x7f326c15d530 2023-12-04T05:39:38.791 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f3273026640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c155d70 con 0x7f326c0724f0 2023-12-04T05:39:38.791 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f3273026640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f326c156330 con 0x7f326c154160 2023-12-04T05:39:38.791 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f32715c3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f326c15d530 0x7f326c0bdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:38.791 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f32715c3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f326c15d530 0x7f326c0bdbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44460/0 (socket says 172.21.15.107:44460) 2023-12-04T05:39:38.791 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f32715c3640 1 -- 172.21.15.107:0/4138720247 learned_addr learned my addr 172.21.15.107:0/4138720247 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:39:38.791 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f3270dc2640 1 --2- 172.21.15.107:0/4138720247 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f326c0724f0 0x7f326c1efc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:38.791 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f32715c3640 1 -- 172.21.15.107:0/4138720247 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f326c154160 msgr2=0x7f326c153510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:38.791 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f326bfff640 1 --2- 172.21.15.107:0/4138720247 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f326c154160 0x7f326c153510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:38.791 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f32715c3640 1 --2- 172.21.15.107:0/4138720247 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f326c154160 0x7f326c153510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:38.791 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f32715c3640 1 -- 172.21.15.107:0/4138720247 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f326c0724f0 msgr2=0x7f326c1efc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:38.791 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f32715c3640 1 --2- 172.21.15.107:0/4138720247 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f326c0724f0 0x7f326c1efc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:38.792 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f32715c3640 1 -- 172.21.15.107:0/4138720247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3258009420 con 0x7f326c15d530 2023-12-04T05:39:38.792 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f3270dc2640 1 --2- 172.21.15.107:0/4138720247 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f326c0724f0 0x7f326c1efc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:39:38.792 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f32715c3640 1 --2- 172.21.15.107:0/4138720247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f326c15d530 0x7f326c0bdbb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f326400b9a0 tx=0x7f326400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:38.792 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f32697fa640 1 -- 172.21.15.107:0/4138720247 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326400cba0 con 0x7f326c15d530 2023-12-04T05:39:38.792 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f32697fa640 1 -- 172.21.15.107:0/4138720247 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326400cd00 con 0x7f326c15d530 2023-12-04T05:39:38.792 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f3273026640 1 -- 172.21.15.107:0/4138720247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f326c0be760 con 0x7f326c15d530 2023-12-04T05:39:38.792 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f32697fa640 1 -- 172.21.15.107:0/4138720247 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3264014680 con 0x7f326c15d530 2023-12-04T05:39:38.792 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f3273026640 1 -- 172.21.15.107:0/4138720247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f326c0bed00 con 0x7f326c15d530 2023-12-04T05:39:38.792 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f32697fa640 1 -- 172.21.15.107:0/4138720247 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3264003c80 con 0x7f326c15d530 2023-12-04T05:39:38.792 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.785+0000 7f32697fa640 1 -- 172.21.15.107:0/4138720247 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f3264004790 con 0x7f326c15d530 2023-12-04T05:39:38.792 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.789+0000 7f3273026640 1 -- 172.21.15.107:0/4138720247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f326c072c40 con 0x7f326c15d530 2023-12-04T05:39:38.793 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.789+0000 7f32697fa640 1 -- 172.21.15.107:0/4138720247 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f3264019070 con 0x7f326c15d530 2023-12-04T05:39:38.850 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.845+0000 7f3273026640 1 -- 172.21.15.107:0/4138720247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f326c067ec0 con 0x7f326c15d530 2023-12-04T05:39:38.851 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.849+0000 7f32697fa640 1 -- 172.21.15.107:0/4138720247 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f3264014890 con 0x7f326c15d530 2023-12-04T05:39:38.851 INFO:teuthology.misc.health.smithi107.stdout: 2023-12-04T05:39:38.851 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"603e856b-31ac-475a-8f35-21ab7faadd48","created":"2023-12-04T05:39:34.921690+0000","modified":"2023-12-04T05:39:37.587639+0000","last_up_change":"0.000000","last_in_change":"2023-12-04T05:39:37.587639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0ab7111c-0ae8-4d41-b6db-e3743614322c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"baef7a1f-8026-4fab-9e70-48465271034e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"90e8cf24-7b54-484a-9d85-b44a9d3dfeab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"54658731-2b9a-4d5e-ab13-7c1be7d34e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9a9dea13-8afe-485d-aa43-45eda691179b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"db840d83-5a7d-468a-8aa2-1997945847c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"38b5f165-bf91-467e-a615-4b4784337689","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e5b3c596-494b-4df3-9437-bc2d52168ac4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T05:39:38.852 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.849+0000 7f3273026640 1 -- 172.21.15.107:0/4138720247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f326c15d530 msgr2=0x7f326c0bdbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:38.852 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.849+0000 7f3273026640 1 --2- 172.21.15.107:0/4138720247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f326c15d530 0x7f326c0bdbb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f326400b9a0 tx=0x7f326400be60 comp rx=0 tx=0).stop 2023-12-04T05:39:38.852 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.849+0000 7f3273026640 1 -- 172.21.15.107:0/4138720247 shutdown_connections 2023-12-04T05:39:38.852 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.849+0000 7f3273026640 1 --2- 172.21.15.107:0/4138720247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f326c15d530 0x7f326c0bdbb0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:38.852 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.849+0000 7f3273026640 1 --2- 172.21.15.107:0/4138720247 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f326c154160 0x7f326c153510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:38.852 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.849+0000 7f3273026640 1 --2- 172.21.15.107:0/4138720247 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f326c0724f0 0x7f326c1efc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:38.852 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.849+0000 7f3273026640 1 -- 172.21.15.107:0/4138720247 >> 172.21.15.107:0/4138720247 conn(0x7f326c06fa00 msgr2=0x7f326c0c8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:38.853 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.849+0000 7f3273026640 1 -- 172.21.15.107:0/4138720247 shutdown_connections 2023-12-04T05:39:38.853 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:38.849+0000 7f3273026640 1 -- 172.21.15.107:0/4138720247 wait complete. 2023-12-04T05:39:38.864 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:38.861+0000 7fdb3fdd0180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-04T05:39:38.867 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-04T05:39:38.906 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:38.905+0000 7f7cb9017180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-04T05:39:38.982 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:38.981+0000 7f7cb9017180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-04T05:39:39.036 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:39.033+0000 7fdb3fdd0180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-04T05:39:39.109 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:39.105+0000 7fdb3fdd0180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-04T05:39:39.152 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:39.149+0000 7f7cb9017180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-04T05:39:39.225 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:39.221+0000 7f7cb9017180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-04T05:39:39.396 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:39.393+0000 7fdb3fdd0180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-04T05:39:39.455 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:39.453+0000 7fdb3fdd0180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-04T05:39:39.515 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:39.513+0000 7f7cb9017180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-04T05:39:39.578 INFO:tasks.ceph.mgr.x.smithi195.stderr:2023-12-04T05:39:39.577+0000 7f7cb9017180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-04T05:39:40.500 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:40.496+0000 7fdae7559640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-12-04T05:39:40.500 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:39:40.496+0000 7fdae7559640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-12-04T05:39:42.057 INFO:tasks.ceph.osd.4.smithi195.stderr:2023-12-04T05:39:42.053+0000 7f20f6823640 -1 osd.4 0 log_to_monitors true 2023-12-04T05:39:42.063 INFO:tasks.ceph.osd.7.smithi195.stderr:2023-12-04T05:39:42.061+0000 7f82fdce9640 -1 osd.7 0 log_to_monitors true 2023-12-04T05:39:42.099 INFO:tasks.ceph.osd.1.smithi107.stderr:2023-12-04T05:39:42.096+0000 7f4957ccb640 -1 osd.1 0 log_to_monitors true 2023-12-04T05:39:42.108 INFO:tasks.ceph.osd.6.smithi195.stderr:2023-12-04T05:39:42.105+0000 7fd574750640 -1 osd.6 0 log_to_monitors true 2023-12-04T05:39:42.191 INFO:tasks.ceph.osd.5.smithi195.stderr:2023-12-04T05:39:42.189+0000 7f2fc3c87640 -1 osd.5 0 log_to_monitors true 2023-12-04T05:39:42.226 INFO:tasks.ceph.osd.0.smithi107.stderr:2023-12-04T05:39:42.220+0000 7fde5bdaf640 -1 osd.0 0 log_to_monitors true 2023-12-04T05:39:42.241 INFO:tasks.ceph.osd.2.smithi107.stderr:2023-12-04T05:39:42.236+0000 7fad6cc89640 -1 osd.2 0 log_to_monitors true 2023-12-04T05:39:42.255 INFO:tasks.ceph.osd.3.smithi107.stderr:2023-12-04T05:39:42.252+0000 7f775c774640 -1 osd.3 0 log_to_monitors true 2023-12-04T05:39:45.231 INFO:tasks.ceph.osd.4.smithi195.stderr:2023-12-04T05:39:45.229+0000 7f20f1fa9640 -1 osd.4 0 waiting for initial osdmap 2023-12-04T05:39:45.263 INFO:tasks.ceph.osd.4.smithi195.stderr:2023-12-04T05:39:45.261+0000 7f20ea5f6640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T05:39:45.345 INFO:tasks.ceph.osd.5.smithi195.stderr:2023-12-04T05:39:45.341+0000 7f2fbf40d640 -1 osd.5 0 waiting for initial osdmap 2023-12-04T05:39:45.371 INFO:tasks.ceph.osd.5.smithi195.stderr:2023-12-04T05:39:45.369+0000 7f2fb7a5a640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T05:39:45.421 INFO:tasks.ceph.osd.7.smithi195.stderr:2023-12-04T05:39:45.417+0000 7f82f946f640 -1 osd.7 0 waiting for initial osdmap 2023-12-04T05:39:45.446 INFO:tasks.ceph.osd.6.smithi195.stderr:2023-12-04T05:39:45.441+0000 7fd57170c640 -1 osd.6 0 waiting for initial osdmap 2023-12-04T05:39:45.447 INFO:tasks.ceph.osd.7.smithi195.stderr:2023-12-04T05:39:45.445+0000 7f82f1abc640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T05:39:45.463 INFO:tasks.ceph.osd.6.smithi195.stderr:2023-12-04T05:39:45.461+0000 7fd568523640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T05:39:45.673 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-04T05:39:45.840 INFO:tasks.ceph.osd.2.smithi107.stderr:2023-12-04T05:39:45.836+0000 7fad6840f640 -1 osd.2 0 waiting for initial osdmap 2023-12-04T05:39:45.845 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.840+0000 7f5d5eab8640 1 Processor -- start 2023-12-04T05:39:45.845 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.840+0000 7f5d5eab8640 1 -- start start 2023-12-04T05:39:45.845 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.840+0000 7f5d5eab8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5d58150ed0 0x7f5d5815dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:45.846 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.840+0000 7f5d5eab8640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5d5806a640 0x7f5d5806aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:45.846 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.840+0000 7f5d5eab8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d580c0ec0 con 0x7f5d58066450 2023-12-04T05:39:45.846 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.840+0000 7f5d5eab8640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d5806ce40 con 0x7f5d58150ed0 2023-12-04T05:39:45.846 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.840+0000 7f5d5eab8640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5806b030 con 0x7f5d5806a640 2023-12-04T05:39:45.846 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.840+0000 7f5d5d055640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5d5806a640 0x7f5d5806aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:45.847 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.840+0000 7f5d5d055640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5d5806a640 0x7f5d5806aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.107:56554/0 (socket says 172.21.15.107:56554) 2023-12-04T05:39:45.847 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.840+0000 7f5d5d055640 1 -- 172.21.15.107:0/1025118762 learned_addr learned my addr 172.21.15.107:0/1025118762 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:39:45.847 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.840+0000 7f5d5d055640 1 -- 172.21.15.107:0/1025118762 >> v2:172.21.15.107:3300/0 conn(0x7f5d58150ed0 msgr2=0x7f5d5815dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:45.847 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.840+0000 7f5d5d055640 1 --2- 172.21.15.107:0/1025118762 >> v2:172.21.15.107:3300/0 conn(0x7f5d58150ed0 0x7f5d5815dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:45.847 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.840+0000 7f5d5d055640 1 -- 172.21.15.107:0/1025118762 >> v1:172.21.15.195:6789/0 conn(0x7f5d58066450 legacy=0x7f5d580c78b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:39:45.847 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.844+0000 7f5d5d055640 1 -- 172.21.15.107:0/1025118762 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5815e2e0 con 0x7f5d5806a640 2023-12-04T05:39:45.850 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.844+0000 7f5d5d055640 1 --2- 172.21.15.107:0/1025118762 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5d5806a640 0x7f5d5806aaf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5d50009e50 tx=0x7f5d5001c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59ac8f0ab9155cc3 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:45.852 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.848+0000 7f5d577fe640 1 -- 172.21.15.107:0/1025118762 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d500a5790 con 0x7f5d5806a640 2023-12-04T05:39:45.852 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.848+0000 7f5d577fe640 1 -- 172.21.15.107:0/1025118762 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d500a58f0 con 0x7f5d5806a640 2023-12-04T05:39:45.852 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.848+0000 7f5d577fe640 1 -- 172.21.15.107:0/1025118762 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d500a5c80 con 0x7f5d5806a640 2023-12-04T05:39:45.854 INFO:tasks.ceph.osd.1.smithi107.stderr:2023-12-04T05:39:45.848+0000 7f4954c87640 -1 osd.1 0 waiting for initial osdmap 2023-12-04T05:39:45.855 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.852+0000 7f5d5eab8640 1 -- 172.21.15.107:0/1025118762 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5d5806a640 msgr2=0x7f5d5806aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:45.855 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.852+0000 7f5d5eab8640 1 --2- 172.21.15.107:0/1025118762 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5d5806a640 0x7f5d5806aaf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5d50009e50 tx=0x7f5d5001c910 comp rx=0 tx=0).stop 2023-12-04T05:39:45.858 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.852+0000 7f5d5eab8640 1 -- 172.21.15.107:0/1025118762 shutdown_connections 2023-12-04T05:39:45.859 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.852+0000 7f5d5eab8640 1 --2- 172.21.15.107:0/1025118762 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5d5806a640 0x7f5d5806aaf0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:45.859 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.852+0000 7f5d5eab8640 1 --2- 172.21.15.107:0/1025118762 >> v2:172.21.15.107:3300/0 conn(0x7f5d58150ed0 0x7f5d5815dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:45.859 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.852+0000 7f5d5eab8640 1 -- 172.21.15.107:0/1025118762 >> 172.21.15.107:0/1025118762 conn(0x7f5d5806dee0 msgr2=0x7f5d5806e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:45.859 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.852+0000 7f5d5eab8640 1 -- 172.21.15.107:0/1025118762 shutdown_connections 2023-12-04T05:39:45.859 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.852+0000 7f5d5eab8640 1 -- 172.21.15.107:0/1025118762 wait complete. 2023-12-04T05:39:45.859 INFO:tasks.ceph.osd.3.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f7757efa640 -1 osd.3 0 waiting for initial osdmap 2023-12-04T05:39:45.860 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d5eab8640 1 Processor -- start 2023-12-04T05:39:45.860 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d5eab8640 1 -- start start 2023-12-04T05:39:45.861 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d5eab8640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5d58066450 0x7f5d5815afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:45.861 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d5eab8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d5806a640 0x7f5d581f3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:45.861 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d5eab8640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d58150ed0 0x7f5d581475a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:45.861 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d5eab8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5815c9a0 con 0x7f5d5806a640 2023-12-04T05:39:45.862 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d5eab8640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5815cc60 con 0x7f5d58150ed0 2023-12-04T05:39:45.862 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d5eab8640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5815cf40 con 0x7f5d58066450 2023-12-04T05:39:45.862 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d57fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d5806a640 0x7f5d581f3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:45.862 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d57fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d5806a640 0x7f5d581f3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44570/0 (socket says 172.21.15.107:44570) 2023-12-04T05:39:45.862 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d57fff640 1 -- 172.21.15.107:0/1966132205 learned_addr learned my addr 172.21.15.107:0/1966132205 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:39:45.862 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d5c854640 1 --2- 172.21.15.107:0/1966132205 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5d58066450 0x7f5d5815afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:45.862 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d5d055640 1 --2- 172.21.15.107:0/1966132205 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d58150ed0 0x7f5d581475a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:45.862 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d57fff640 1 -- 172.21.15.107:0/1966132205 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5d58066450 msgr2=0x7f5d5815afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:45.862 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d57fff640 1 --2- 172.21.15.107:0/1966132205 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5d58066450 0x7f5d5815afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:45.862 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d57fff640 1 -- 172.21.15.107:0/1966132205 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d58150ed0 msgr2=0x7f5d581475a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:45.862 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d57fff640 1 --2- 172.21.15.107:0/1966132205 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d58150ed0 0x7f5d581475a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:45.863 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d57fff640 1 -- 172.21.15.107:0/1966132205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d50009b30 con 0x7f5d5806a640 2023-12-04T05:39:45.864 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d57fff640 1 --2- 172.21.15.107:0/1966132205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d5806a640 0x7f5d581f3210 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5d4400b3e0 tx=0x7f5d4400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:45.864 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d557fa640 1 -- 172.21.15.107:0/1966132205 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d44011830 con 0x7f5d5806a640 2023-12-04T05:39:45.864 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d557fa640 1 -- 172.21.15.107:0/1966132205 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d44011990 con 0x7f5d5806a640 2023-12-04T05:39:45.864 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d5eab8640 1 -- 172.21.15.107:0/1966132205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d581f2a10 con 0x7f5d5806a640 2023-12-04T05:39:45.864 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.856+0000 7f5d557fa640 1 -- 172.21.15.107:0/1966132205 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d44011d60 con 0x7f5d5806a640 2023-12-04T05:39:45.864 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.860+0000 7f5d557fa640 1 -- 172.21.15.107:0/1966132205 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 4) v1 ==== 86114+0+0 (secure 0 0 0) 0x7f5d4401a400 con 0x7f5d5806a640 2023-12-04T05:39:45.864 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.860+0000 7f5d5eab8640 1 -- 172.21.15.107:0/1966132205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d581f26c0 con 0x7f5d5806a640 2023-12-04T05:39:45.864 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.860+0000 7f5d557fa640 1 --2- 172.21.15.107:0/1966132205 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5d3c066840 0x7f5d3c068d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:45.864 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.860+0000 7f5d5c854640 1 --2- 172.21.15.107:0/1966132205 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5d3c066840 0x7f5d3c068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:45.865 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.860+0000 7f5d5c854640 1 --2- 172.21.15.107:0/1966132205 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5d3c066840 0x7f5d3c068d00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c006fd0 tx=0x7f5d4c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:45.865 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.860+0000 7f5d557fa640 1 -- 172.21.15.107:0/1966132205 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f5d44054de0 con 0x7f5d5806a640 2023-12-04T05:39:45.865 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.860+0000 7f5d42ffd640 1 -- 172.21.15.107:0/1966132205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d30005260 con 0x7f5d5806a640 2023-12-04T05:39:45.871 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:45.868+0000 7f5d557fa640 1 -- 172.21.15.107:0/1966132205 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5d44050a20 con 0x7f5d5806a640 2023-12-04T05:39:45.875 INFO:tasks.ceph.osd.2.smithi107.stderr:2023-12-04T05:39:45.872+0000 7fad60a5c640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T05:39:45.892 INFO:tasks.ceph.osd.1.smithi107.stderr:2023-12-04T05:39:45.888+0000 7f494ba9e640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T05:39:45.896 INFO:tasks.ceph.osd.3.smithi107.stderr:2023-12-04T05:39:45.892+0000 7f7750547640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T05:39:45.941 INFO:tasks.ceph.osd.0.smithi107.stderr:2023-12-04T05:39:45.936+0000 7fde57535640 -1 osd.0 0 waiting for initial osdmap 2023-12-04T05:39:45.964 INFO:tasks.ceph.osd.0.smithi107.stderr:2023-12-04T05:39:45.960+0000 7fde4fb82640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T05:39:46.019 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:46.016+0000 7f5d42ffd640 1 -- 172.21.15.107:0/1966132205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d300057e0 con 0x7f5d5806a640 2023-12-04T05:39:46.020 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:46.016+0000 7f5d557fa640 1 -- 172.21.15.107:0/1966132205 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f5d44054040 con 0x7f5d5806a640 2023-12-04T05:39:46.020 INFO:teuthology.misc.health.smithi107.stdout: 2023-12-04T05:39:46.021 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":12,"fsid":"603e856b-31ac-475a-8f35-21ab7faadd48","created":"2023-12-04T05:39:34.921690+0000","modified":"2023-12-04T05:39:43.501802+0000","last_up_change":"0.000000","last_in_change":"2023-12-04T05:39:37.587639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0ab7111c-0ae8-4d41-b6db-e3743614322c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"baef7a1f-8026-4fab-9e70-48465271034e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"90e8cf24-7b54-484a-9d85-b44a9d3dfeab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"54658731-2b9a-4d5e-ab13-7c1be7d34e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9a9dea13-8afe-485d-aa43-45eda691179b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"db840d83-5a7d-468a-8aa2-1997945847c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"38b5f165-bf91-467e-a615-4b4784337689","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e5b3c596-494b-4df3-9437-bc2d52168ac4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T05:39:46.022 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:46.016+0000 7f5d42ffd640 1 -- 172.21.15.107:0/1966132205 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5d3c066840 msgr2=0x7f5d3c068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:46.023 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:46.016+0000 7f5d42ffd640 1 --2- 172.21.15.107:0/1966132205 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5d3c066840 0x7f5d3c068d00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c006fd0 tx=0x7f5d4c008040 comp rx=0 tx=0).stop 2023-12-04T05:39:46.023 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:46.016+0000 7f5d42ffd640 1 -- 172.21.15.107:0/1966132205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d5806a640 msgr2=0x7f5d581f3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:46.023 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:46.016+0000 7f5d42ffd640 1 --2- 172.21.15.107:0/1966132205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d5806a640 0x7f5d581f3210 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5d4400b3e0 tx=0x7f5d4400b8a0 comp rx=0 tx=0).stop 2023-12-04T05:39:46.023 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:46.016+0000 7f5d42ffd640 1 -- 172.21.15.107:0/1966132205 shutdown_connections 2023-12-04T05:39:46.023 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:46.016+0000 7f5d42ffd640 1 --2- 172.21.15.107:0/1966132205 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5d3c066840 0x7f5d3c068d00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:46.023 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:46.016+0000 7f5d42ffd640 1 --2- 172.21.15.107:0/1966132205 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d58150ed0 0x7f5d581475a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:46.023 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:46.016+0000 7f5d42ffd640 1 --2- 172.21.15.107:0/1966132205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d5806a640 0x7f5d581f3210 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:46.023 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:46.016+0000 7f5d42ffd640 1 --2- 172.21.15.107:0/1966132205 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5d58066450 0x7f5d5815afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:46.023 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:46.016+0000 7f5d42ffd640 1 -- 172.21.15.107:0/1966132205 >> 172.21.15.107:0/1966132205 conn(0x7f5d5806dee0 msgr2=0x7f5d580ca680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:46.023 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:46.016+0000 7f5d42ffd640 1 -- 172.21.15.107:0/1966132205 shutdown_connections 2023-12-04T05:39:46.023 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:46.016+0000 7f5d42ffd640 1 -- 172.21.15.107:0/1966132205 wait complete. 2023-12-04T05:39:46.036 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-04T05:39:52.843 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-04T05:39:52.964 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f236a6cd640 1 Processor -- start 2023-12-04T05:39:52.964 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f236a6cd640 1 -- start start 2023-12-04T05:39:52.965 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f236a6cd640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f23640c4d80 0x7f23641580b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:52.965 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f236a6cd640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2364141120 0x7f23641585f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:52.965 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f236a6cd640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f236415b1a0 con 0x7f23640c5160 2023-12-04T05:39:52.965 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f236a6cd640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2364157c90 con 0x7f2364141120 2023-12-04T05:39:52.965 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f236a6cd640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f236415b400 con 0x7f23640c4d80 2023-12-04T05:39:52.967 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f23637fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f23640c5160 0x7f2364140a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53340/0 (socket says 172.21.15.107:53340) 2023-12-04T05:39:52.967 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f23637fe640 1 -- 172.21.15.107:0/2338364848 learned_addr learned my addr 172.21.15.107:0/2338364848 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:39:52.967 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f2368c6a640 1 --2- 172.21.15.107:0/2338364848 >> v2:172.21.15.195:3300/0 conn(0x7f2364141120 0x7f23641585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:52.967 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f2363fff640 1 --2- 172.21.15.107:0/2338364848 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f23640c4d80 0x7f23641580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:52.967 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f2362ffd640 1 -- 172.21.15.107:0/2338364848 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 403535331 0 0) 0x7f236415b1a0 con 0x7f23640c5160 2023-12-04T05:39:52.967 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f2362ffd640 1 -- 172.21.15.107:0/2338364848 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f234c003530 con 0x7f23640c5160 2023-12-04T05:39:52.967 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f2362ffd640 1 -- 172.21.15.107:0/2338364848 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3775360736 0 0) 0x7f234c003530 con 0x7f23640c5160 2023-12-04T05:39:52.967 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f2362ffd640 1 -- 172.21.15.107:0/2338364848 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f23640c4d80 msgr2=0x7f23641580b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:52.967 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f2362ffd640 1 --2- 172.21.15.107:0/2338364848 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f23640c4d80 0x7f23641580b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:52.967 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f2362ffd640 1 -- 172.21.15.107:0/2338364848 >> v2:172.21.15.195:3300/0 conn(0x7f2364141120 msgr2=0x7f23641585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:52.967 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f2362ffd640 1 --2- 172.21.15.107:0/2338364848 >> v2:172.21.15.195:3300/0 conn(0x7f2364141120 0x7f23641585f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:52.968 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f2362ffd640 1 -- 172.21.15.107:0/2338364848 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f236415b6e0 con 0x7f23640c5160 2023-12-04T05:39:52.968 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f2362ffd640 1 -- 172.21.15.107:0/2338364848 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f2358001680 con 0x7f23640c5160 2023-12-04T05:39:52.968 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f2362ffd640 1 -- 172.21.15.107:0/2338364848 >> v1:172.21.15.107:6789/0 conn(0x7f23640c5160 legacy=0x7f2364140a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:52.968 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f2362ffd640 1 --2- 172.21.15.107:0/2338364848 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f234c003980 0x7f234c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:52.968 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f2362ffd640 1 --2- 172.21.15.107:0/2338364848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f234c0047d0 0x7f234c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:52.968 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f2362ffd640 1 --2- 172.21.15.107:0/2338364848 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f23640c4d80 0x7f234c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:52.968 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f2362ffd640 1 -- 172.21.15.107:0/2338364848 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2364157c90 con 0x7f234c0047d0 2023-12-04T05:39:52.968 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f2362ffd640 1 -- 172.21.15.107:0/2338364848 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236415b400 con 0x7f234c003980 2023-12-04T05:39:52.968 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f2362ffd640 1 -- 172.21.15.107:0/2338364848 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f234c00b990 con 0x7f23640c4d80 2023-12-04T05:39:52.968 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f23637fe640 1 --2- 172.21.15.107:0/2338364848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f234c0047d0 0x7f234c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:52.968 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f2363fff640 1 --2- 172.21.15.107:0/2338364848 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f234c003980 0x7f234c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:52.968 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f23637fe640 1 -- 172.21.15.107:0/2338364848 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f23640c4d80 msgr2=0x7f234c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:52.968 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f23637fe640 1 --2- 172.21.15.107:0/2338364848 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f23640c4d80 0x7f234c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:52.968 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f23637fe640 1 -- 172.21.15.107:0/2338364848 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f234c003980 msgr2=0x7f234c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:52.968 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f23637fe640 1 --2- 172.21.15.107:0/2338364848 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f234c003980 0x7f234c003da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:52.969 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.960+0000 7f23637fe640 1 -- 172.21.15.107:0/2338364848 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f234c003530 con 0x7f234c0047d0 2023-12-04T05:39:52.970 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.964+0000 7f23637fe640 1 --2- 172.21.15.107:0/2338364848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f234c0047d0 0x7f234c004c50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f235800c240 tx=0x7f235801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=694ad24fb9412dba server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:52.970 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.964+0000 7f2362ffd640 1 -- 172.21.15.107:0/2338364848 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23580058e0 con 0x7f234c0047d0 2023-12-04T05:39:52.970 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.964+0000 7f2362ffd640 1 -- 172.21.15.107:0/2338364848 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2358005a40 con 0x7f234c0047d0 2023-12-04T05:39:52.970 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.964+0000 7f2362ffd640 1 -- 172.21.15.107:0/2338364848 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2358005e10 con 0x7f234c0047d0 2023-12-04T05:39:52.970 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.964+0000 7f236a6cd640 1 -- 172.21.15.107:0/2338364848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f234c0047d0 msgr2=0x7f234c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:52.970 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.964+0000 7f236a6cd640 1 --2- 172.21.15.107:0/2338364848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f234c0047d0 0x7f234c004c50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f235800c240 tx=0x7f235801cc60 comp rx=0 tx=0).stop 2023-12-04T05:39:52.970 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.964+0000 7f2368c6a640 1 -- 172.21.15.107:0/2338364848 reap_dead start 2023-12-04T05:39:52.970 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.964+0000 7f236a6cd640 1 -- 172.21.15.107:0/2338364848 shutdown_connections 2023-12-04T05:39:52.970 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.964+0000 7f236a6cd640 1 -- 172.21.15.107:0/2338364848 >> 172.21.15.107:0/2338364848 conn(0x7f23640cb410 msgr2=0x7f23640cb820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:52.970 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.964+0000 7f236a6cd640 1 -- 172.21.15.107:0/2338364848 shutdown_connections 2023-12-04T05:39:52.970 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.964+0000 7f236a6cd640 1 -- 172.21.15.107:0/2338364848 wait complete. 2023-12-04T05:39:52.972 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.964+0000 7f236a6cd640 1 Processor -- start 2023-12-04T05:39:52.972 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.964+0000 7f236a6cd640 1 -- start start 2023-12-04T05:39:52.972 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f236a6cd640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f234c0047d0 0x7f23640b8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:52.972 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f236a6cd640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f23640c7580 0x7f23641552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:52.972 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f236a6cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2364155820 0x7f236413bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:52.972 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f236a6cd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2364155ca0 con 0x7f2364155820 2023-12-04T05:39:52.972 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f236a6cd640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23641ee3c0 con 0x7f234c0047d0 2023-12-04T05:39:52.973 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f236a6cd640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23641ed840 con 0x7f23640c7580 2023-12-04T05:39:52.973 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f2368c6a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2364155820 0x7f236413bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:52.973 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f2368c6a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2364155820 0x7f236413bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33788/0 (socket says 172.21.15.107:33788) 2023-12-04T05:39:52.973 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f2368c6a640 1 -- 172.21.15.107:0/152302463 learned_addr learned my addr 172.21.15.107:0/152302463 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:39:52.973 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f23637fe640 1 --2- 172.21.15.107:0/152302463 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f23640c7580 0x7f23641552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:52.973 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f2363fff640 1 --2- 172.21.15.107:0/152302463 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f234c0047d0 0x7f23640b8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:52.973 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f2368c6a640 1 -- 172.21.15.107:0/152302463 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f23640c7580 msgr2=0x7f23641552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:52.973 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f2368c6a640 1 --2- 172.21.15.107:0/152302463 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f23640c7580 0x7f23641552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:52.973 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f2368c6a640 1 -- 172.21.15.107:0/152302463 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f234c0047d0 msgr2=0x7f23640b8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:52.973 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f2368c6a640 1 --2- 172.21.15.107:0/152302463 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f234c0047d0 0x7f23640b8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:52.973 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f2368c6a640 1 -- 172.21.15.107:0/152302463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23641edb20 con 0x7f2364155820 2023-12-04T05:39:52.973 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f2368c6a640 1 --2- 172.21.15.107:0/152302463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2364155820 0x7f236413bcf0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2364141500 tx=0x7f235c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:52.973 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f2360ff9640 1 -- 172.21.15.107:0/152302463 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c010040 con 0x7f2364155820 2023-12-04T05:39:52.973 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f2360ff9640 1 -- 172.21.15.107:0/152302463 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235c00bce0 con 0x7f2364155820 2023-12-04T05:39:52.974 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f2360ff9640 1 -- 172.21.15.107:0/152302463 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c014640 con 0x7f2364155820 2023-12-04T05:39:52.974 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f236a6cd640 1 -- 172.21.15.107:0/152302463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23641ede00 con 0x7f2364155820 2023-12-04T05:39:52.974 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f236a6cd640 1 -- 172.21.15.107:0/152302463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2364154dc0 con 0x7f2364155820 2023-12-04T05:39:52.975 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f236a6cd640 1 -- 172.21.15.107:0/152302463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2364067870 con 0x7f2364155820 2023-12-04T05:39:52.975 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f2360ff9640 1 -- 172.21.15.107:0/152302463 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f235c0147a0 con 0x7f2364155820 2023-12-04T05:39:52.975 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f2360ff9640 1 --2- 172.21.15.107:0/152302463 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f23500668d0 0x7f2350068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:52.975 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f2360ff9640 1 -- 172.21.15.107:0/152302463 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f235c085430 con 0x7f2364155820 2023-12-04T05:39:52.975 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f2363fff640 1 --2- 172.21.15.107:0/152302463 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f23500668d0 0x7f2350068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:52.975 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.968+0000 7f2363fff640 1 --2- 172.21.15.107:0/152302463 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f23500668d0 0x7f2350068d90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2354009850 tx=0x7f2354008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:52.978 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:52.972+0000 7f2360ff9640 1 -- 172.21.15.107:0/152302463 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f235c052aa0 con 0x7f2364155820 2023-12-04T05:39:53.132 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:53.128+0000 7f236a6cd640 1 -- 172.21.15.107:0/152302463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2364140b20 con 0x7f2364155820 2023-12-04T05:39:53.134 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:53.128+0000 7f2360ff9640 1 -- 172.21.15.107:0/152302463 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12886 (secure 0 0 0) 0x7f235c0560c0 con 0x7f2364155820 2023-12-04T05:39:53.134 INFO:teuthology.misc.health.smithi107.stdout: 2023-12-04T05:39:53.134 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":17,"fsid":"603e856b-31ac-475a-8f35-21ab7faadd48","created":"2023-12-04T05:39:34.921690+0000","modified":"2023-12-04T05:39:50.257517+0000","last_up_change":"2023-12-04T05:39:46.235790+0000","last_in_change":"2023-12-04T05:39:37.587639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T05:39:47.567702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ab7111c-0ae8-4d41-b6db-e3743614322c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6817","nonce":768714279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6821","nonce":768714279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6829","nonce":768714279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6825","nonce":768714279}]},"public_addr":"172.21.15.107:6817/768714279","cluster_addr":"172.21.15.107:6821/768714279","heartbeat_back_addr":"172.21.15.107:6829/768714279","heartbeat_front_addr":"172.21.15.107:6825/768714279","state":["exists","up"]},{"osd":1,"uuid":"baef7a1f-8026-4fab-9e70-48465271034e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6801","nonce":2496355981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6803","nonce":2496355981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6807","nonce":2496355981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6805","nonce":2496355981}]},"public_addr":"172.21.15.107:6801/2496355981","cluster_addr":"172.21.15.107:6803/2496355981","heartbeat_back_addr":"172.21.15.107:6807/2496355981","heartbeat_front_addr":"172.21.15.107:6805/2496355981","state":["exists","up"]},{"osd":2,"uuid":"90e8cf24-7b54-484a-9d85-b44a9d3dfeab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6809","nonce":3097298224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6811","nonce":3097298224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6815","nonce":3097298224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6813","nonce":3097298224}]},"public_addr":"172.21.15.107:6809/3097298224","cluster_addr":"172.21.15.107:6811/3097298224","heartbeat_back_addr":"172.21.15.107:6815/3097298224","heartbeat_front_addr":"172.21.15.107:6813/3097298224","state":["exists","up"]},{"osd":3,"uuid":"54658731-2b9a-4d5e-ab13-7c1be7d34e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6820","nonce":3773076671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6824","nonce":3773076671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6831","nonce":3773076671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6828","nonce":3773076671}]},"public_addr":"172.21.15.107:6820/3773076671","cluster_addr":"172.21.15.107:6824/3773076671","heartbeat_back_addr":"172.21.15.107:6831/3773076671","heartbeat_front_addr":"172.21.15.107:6828/3773076671","state":["exists","up"]},{"osd":4,"uuid":"9a9dea13-8afe-485d-aa43-45eda691179b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6801","nonce":2747823542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6803","nonce":2747823542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6807","nonce":2747823542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6805","nonce":2747823542}]},"public_addr":"172.21.15.195:6801/2747823542","cluster_addr":"172.21.15.195:6803/2747823542","heartbeat_back_addr":"172.21.15.195:6807/2747823542","heartbeat_front_addr":"172.21.15.195:6805/2747823542","state":["exists","up"]},{"osd":5,"uuid":"db840d83-5a7d-468a-8aa2-1997945847c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6825","nonce":1484264540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6827","nonce":1484264540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6831","nonce":1484264540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6829","nonce":1484264540}]},"public_addr":"172.21.15.195:6825/1484264540","cluster_addr":"172.21.15.195:6827/1484264540","heartbeat_back_addr":"172.21.15.195:6831/1484264540","heartbeat_front_addr":"172.21.15.195:6829/1484264540","state":["exists","up"]},{"osd":6,"uuid":"38b5f165-bf91-467e-a615-4b4784337689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6811","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6813","nonce":2638471412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6817","nonce":2638471412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6823","nonce":2638471412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6821","nonce":2638471412}]},"public_addr":"172.21.15.195:6813/2638471412","cluster_addr":"172.21.15.195:6817/2638471412","heartbeat_back_addr":"172.21.15.195:6823/2638471412","heartbeat_front_addr":"172.21.15.195:6821/2638471412","state":["exists","up"]},{"osd":7,"uuid":"e5b3c596-494b-4df3-9437-bc2d52168ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6809","nonce":768189614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6812","nonce":768189614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6820","nonce":768189614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6816","nonce":768189614}]},"public_addr":"172.21.15.195:6809/768189614","cluster_addr":"172.21.15.195:6812/768189614","heartbeat_back_addr":"172.21.15.195:6820/768189614","heartbeat_front_addr":"172.21.15.195:6816/768189614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.209077+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.086601+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.221568+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.238524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.059477+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.148422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.079867+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.064916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T05:39:53.136 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:53.132+0000 7f236a6cd640 1 -- 172.21.15.107:0/152302463 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f23500668d0 msgr2=0x7f2350068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:53.136 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:53.132+0000 7f236a6cd640 1 --2- 172.21.15.107:0/152302463 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f23500668d0 0x7f2350068d90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2354009850 tx=0x7f2354008040 comp rx=0 tx=0).stop 2023-12-04T05:39:53.136 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:53.132+0000 7f236a6cd640 1 -- 172.21.15.107:0/152302463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2364155820 msgr2=0x7f236413bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:53.136 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:53.132+0000 7f236a6cd640 1 --2- 172.21.15.107:0/152302463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2364155820 0x7f236413bcf0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2364141500 tx=0x7f235c00b480 comp rx=0 tx=0).stop 2023-12-04T05:39:53.136 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:53.132+0000 7f236a6cd640 1 -- 172.21.15.107:0/152302463 shutdown_connections 2023-12-04T05:39:53.137 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:53.132+0000 7f236a6cd640 1 --2- 172.21.15.107:0/152302463 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f23500668d0 0x7f2350068d90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.137 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:53.132+0000 7f236a6cd640 1 --2- 172.21.15.107:0/152302463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2364155820 0x7f236413bcf0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.137 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:53.132+0000 7f236a6cd640 1 --2- 172.21.15.107:0/152302463 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f23640c7580 0x7f23641552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.137 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:53.132+0000 7f236a6cd640 1 --2- 172.21.15.107:0/152302463 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f234c0047d0 0x7f23640b8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.137 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:53.132+0000 7f236a6cd640 1 -- 172.21.15.107:0/152302463 >> 172.21.15.107:0/152302463 conn(0x7f23640cb410 msgr2=0x7f2364071f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:53.137 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:53.132+0000 7f236a6cd640 1 -- 172.21.15.107:0/152302463 shutdown_connections 2023-12-04T05:39:53.137 INFO:teuthology.misc.health.smithi107.stderr:2023-12-04T05:39:53.132+0000 7f236a6cd640 1 -- 172.21.15.107:0/152302463 wait complete. 2023-12-04T05:39:53.152 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-12-04T05:39:53.152 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-04T05:39:53.152 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-04T05:39:53.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-04T05:39:53.155 INFO:tasks.ceph.mds.a:Started 2023-12-04T05:39:53.155 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-04T05:39:53.155 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-04T05:39:53.157 INFO:tasks.ceph.mds.c:Started 2023-12-04T05:39:53.157 INFO:tasks.ceph.mds.e:Restarting daemon 2023-12-04T05:39:53.157 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-12-04T05:39:53.159 INFO:tasks.ceph.mds.e:Started 2023-12-04T05:39:53.159 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-04T05:39:53.159 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-04T05:39:53.162 INFO:tasks.ceph.mds.b:Started 2023-12-04T05:39:53.162 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-04T05:39:53.163 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-04T05:39:53.165 INFO:tasks.ceph.mds.d:Started 2023-12-04T05:39:53.165 INFO:tasks.ceph.mds.f:Restarting daemon 2023-12-04T05:39:53.165 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-12-04T05:39:53.167 INFO:tasks.ceph.mds.f:Started 2023-12-04T05:39:53.167 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-12-04T05:39:53.168 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T05:39:53.237 INFO:tasks.ceph.mds.e.smithi107.stdout:starting mds.e at 2023-12-04T05:39:53.238 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-12-04T05:39:53.246 INFO:tasks.ceph.mds.d.smithi195.stdout:starting mds.d at 2023-12-04T05:39:53.247 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-12-04T05:39:53.247 INFO:tasks.ceph.mds.b.smithi195.stdout:starting mds.b at 2023-12-04T05:39:53.247 INFO:tasks.ceph.mds.f.smithi195.stdout:starting mds.f at 2023-12-04T05:39:53.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.284+0000 7f660bc34640 1 Processor -- start 2023-12-04T05:39:53.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.284+0000 7f660bc34640 1 -- start start 2023-12-04T05:39:53.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.284+0000 7f660bc34640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f66040ca2d0 0x7f6604072570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.284+0000 7f660bc34640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6604072ab0 0x7f6604151960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.284+0000 7f660bc34640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66040c9570 con 0x7f66040ca6b0 2023-12-04T05:39:53.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.284+0000 7f660bc34640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6604073290 con 0x7f66040ca2d0 2023-12-04T05:39:53.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.284+0000 7f660bc34640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66040c8c80 con 0x7f6604072ab0 2023-12-04T05:39:53.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.284+0000 7f66099d0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f66040ca2d0 0x7f6604072570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.284+0000 7f66091cf640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f66040ca6b0 0x7f66040cd510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53352/0 (socket says 172.21.15.107:53352) 2023-12-04T05:39:53.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.284+0000 7f66099d0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f66040ca2d0 0x7f6604072570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33832/0 (socket says 172.21.15.107:33832) 2023-12-04T05:39:53.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.284+0000 7f66091cf640 1 -- 172.21.15.107:0/381643443 learned_addr learned my addr 172.21.15.107:0/381643443 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:39:53.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.284+0000 7f660a1d1640 1 --2- 172.21.15.107:0/381643443 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6604072ab0 0x7f6604151960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.284+0000 7f66099d0640 1 -- 172.21.15.107:0/381643443 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6604072ab0 msgr2=0x7f6604151960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:53.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.284+0000 7f66099d0640 1 --2- 172.21.15.107:0/381643443 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6604072ab0 0x7f6604151960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.284+0000 7f66099d0640 1 -- 172.21.15.107:0/381643443 >> v1:172.21.15.107:6789/0 conn(0x7f66040ca6b0 legacy=0x7f66040cd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:53.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.284+0000 7f66089ce640 1 -- 172.21.15.107:0/381643443 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345757125 0 0) 0x7f66040c9570 con 0x7f66040ca6b0 2023-12-04T05:39:53.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.284+0000 7f66099d0640 1 -- 172.21.15.107:0/381643443 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6604072fb0 con 0x7f66040ca2d0 2023-12-04T05:39:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.288+0000 7f66099d0640 1 --2- 172.21.15.107:0/381643443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66040ca2d0 0x7f6604072570 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f66000097f0 tx=0x7f660001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b239c1f713a51f server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.288+0000 7f66089ce640 1 -- 172.21.15.107:0/381643443 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66000a8970 con 0x7f66040ca2d0 2023-12-04T05:39:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.288+0000 7f66089ce640 1 -- 172.21.15.107:0/381643443 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66000a8ad0 con 0x7f66040ca2d0 2023-12-04T05:39:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.288+0000 7f66089ce640 1 -- 172.21.15.107:0/381643443 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66000a8e60 con 0x7f66040ca2d0 2023-12-04T05:39:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.288+0000 7f660bc34640 1 -- 172.21.15.107:0/381643443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66040ca2d0 msgr2=0x7f6604072570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.288+0000 7f660bc34640 1 --2- 172.21.15.107:0/381643443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66040ca2d0 0x7f6604072570 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f66000097f0 tx=0x7f660001ec70 comp rx=0 tx=0).stop 2023-12-04T05:39:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.288+0000 7f660bc34640 1 -- 172.21.15.107:0/381643443 shutdown_connections 2023-12-04T05:39:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.288+0000 7f660bc34640 1 --2- 172.21.15.107:0/381643443 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6604072ab0 0x7f6604151960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.288+0000 7f660bc34640 1 --2- 172.21.15.107:0/381643443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66040ca2d0 0x7f6604072570 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.288+0000 7f660bc34640 1 -- 172.21.15.107:0/381643443 >> 172.21.15.107:0/381643443 conn(0x7f66040663a0 msgr2=0x7f66040667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:53.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.288+0000 7f660bc34640 1 -- 172.21.15.107:0/381643443 shutdown_connections 2023-12-04T05:39:53.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.288+0000 7f660bc34640 1 -- 172.21.15.107:0/381643443 wait complete. 2023-12-04T05:39:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.288+0000 7f660bc34640 1 Processor -- start 2023-12-04T05:39:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.288+0000 7f660bc34640 1 -- start start 2023-12-04T05:39:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f660bc34640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6604072ab0 0x7f6604150e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f660bc34640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66040ca2d0 0x7f66040cc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f660bc34640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6604144440 0x7f66041f11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f660bc34640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66041f0d90 con 0x7f66040ca2d0 2023-12-04T05:39:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f660bc34640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66041513b0 con 0x7f6604072ab0 2023-12-04T05:39:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f660bc34640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6604158dd0 con 0x7f6604144440 2023-12-04T05:39:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f66091cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66040ca2d0 0x7f66040cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f66091cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66040ca2d0 0x7f66040cc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33840/0 (socket says 172.21.15.107:33840) 2023-12-04T05:39:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f66091cf640 1 -- 172.21.15.107:0/382262542 learned_addr learned my addr 172.21.15.107:0/382262542 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:39:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f66099d0640 1 --2- 172.21.15.107:0/382262542 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6604072ab0 0x7f6604150e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f660a1d1640 1 --2- 172.21.15.107:0/382262542 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6604144440 0x7f66041f11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f66091cf640 1 -- 172.21.15.107:0/382262542 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6604144440 msgr2=0x7f66041f11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f66091cf640 1 --2- 172.21.15.107:0/382262542 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6604144440 0x7f66041f11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f66091cf640 1 -- 172.21.15.107:0/382262542 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6604072ab0 msgr2=0x7f6604150e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f66091cf640 1 --2- 172.21.15.107:0/382262542 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6604072ab0 0x7f6604150e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f66091cf640 1 -- 172.21.15.107:0/382262542 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6600009420 con 0x7f66040ca2d0 2023-12-04T05:39:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f66091cf640 1 --2- 172.21.15.107:0/382262542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66040ca2d0 0x7f66040cc630 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f65f400c480 tx=0x7f65f400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f65fa7fc640 1 -- 172.21.15.107:0/382262542 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f40074e0 con 0x7f66040ca2d0 2023-12-04T05:39:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f65fa7fc640 1 -- 172.21.15.107:0/382262542 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f4007640 con 0x7f66040ca2d0 2023-12-04T05:39:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f660bc34640 1 -- 172.21.15.107:0/382262542 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6604159070 con 0x7f66040ca2d0 2023-12-04T05:39:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f65fa7fc640 1 -- 172.21.15.107:0/382262542 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f40049f0 con 0x7f66040ca2d0 2023-12-04T05:39:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f660bc34640 1 -- 172.21.15.107:0/382262542 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6604159630 con 0x7f66040ca2d0 2023-12-04T05:39:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f660bc34640 1 -- 172.21.15.107:0/382262542 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66041f0f00 con 0x7f66040ca2d0 2023-12-04T05:39:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f65fa7fc640 1 -- 172.21.15.107:0/382262542 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f65f4004b50 con 0x7f66040ca2d0 2023-12-04T05:39:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f65fa7fc640 1 --2- 172.21.15.107:0/382262542 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f65e80668d0 0x7f65e8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f65fa7fc640 1 -- 172.21.15.107:0/382262542 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f65f40835c0 con 0x7f66040ca2d0 2023-12-04T05:39:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.292+0000 7f66099d0640 1 --2- 172.21.15.107:0/382262542 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f65e80668d0 0x7f65e8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.296+0000 7f66099d0640 1 --2- 172.21.15.107:0/382262542 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f65e80668d0 0x7f65e8068d90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f66000b2000 tx=0x7f66000aa070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:53.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.300+0000 7f65fa7fc640 1 -- 172.21.15.107:0/382262542 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f66041f0f00 con 0x7f66040ca2d0 2023-12-04T05:39:53.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.448+0000 7f660bc34640 1 -- 172.21.15.107:0/382262542 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66041f0ed0 con 0x7f66040ca2d0 2023-12-04T05:39:53.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.448+0000 7f65fa7fc640 1 -- 172.21.15.107:0/382262542 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12886 (secure 0 0 0) 0x7f65f4018020 con 0x7f66040ca2d0 2023-12-04T05:39:53.455 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:53.455 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"fsid":"603e856b-31ac-475a-8f35-21ab7faadd48","created":"2023-12-04T05:39:34.921690+0000","modified":"2023-12-04T05:39:50.257517+0000","last_up_change":"2023-12-04T05:39:46.235790+0000","last_in_change":"2023-12-04T05:39:37.587639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T05:39:47.567702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ab7111c-0ae8-4d41-b6db-e3743614322c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6817","nonce":768714279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6821","nonce":768714279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6829","nonce":768714279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6825","nonce":768714279}]},"public_addr":"172.21.15.107:6817/768714279","cluster_addr":"172.21.15.107:6821/768714279","heartbeat_back_addr":"172.21.15.107:6829/768714279","heartbeat_front_addr":"172.21.15.107:6825/768714279","state":["exists","up"]},{"osd":1,"uuid":"baef7a1f-8026-4fab-9e70-48465271034e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6801","nonce":2496355981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6803","nonce":2496355981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6807","nonce":2496355981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6805","nonce":2496355981}]},"public_addr":"172.21.15.107:6801/2496355981","cluster_addr":"172.21.15.107:6803/2496355981","heartbeat_back_addr":"172.21.15.107:6807/2496355981","heartbeat_front_addr":"172.21.15.107:6805/2496355981","state":["exists","up"]},{"osd":2,"uuid":"90e8cf24-7b54-484a-9d85-b44a9d3dfeab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6809","nonce":3097298224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6811","nonce":3097298224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6815","nonce":3097298224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6813","nonce":3097298224}]},"public_addr":"172.21.15.107:6809/3097298224","cluster_addr":"172.21.15.107:6811/3097298224","heartbeat_back_addr":"172.21.15.107:6815/3097298224","heartbeat_front_addr":"172.21.15.107:6813/3097298224","state":["exists","up"]},{"osd":3,"uuid":"54658731-2b9a-4d5e-ab13-7c1be7d34e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6820","nonce":3773076671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6824","nonce":3773076671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6831","nonce":3773076671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6828","nonce":3773076671}]},"public_addr":"172.21.15.107:6820/3773076671","cluster_addr":"172.21.15.107:6824/3773076671","heartbeat_back_addr":"172.21.15.107:6831/3773076671","heartbeat_front_addr":"172.21.15.107:6828/3773076671","state":["exists","up"]},{"osd":4,"uuid":"9a9dea13-8afe-485d-aa43-45eda691179b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6801","nonce":2747823542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6803","nonce":2747823542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6807","nonce":2747823542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6805","nonce":2747823542}]},"public_addr":"172.21.15.195:6801/2747823542","cluster_addr":"172.21.15.195:6803/2747823542","heartbeat_back_addr":"172.21.15.195:6807/2747823542","heartbeat_front_addr":"172.21.15.195:6805/2747823542","state":["exists","up"]},{"osd":5,"uuid":"db840d83-5a7d-468a-8aa2-1997945847c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6825","nonce":1484264540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6827","nonce":1484264540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6831","nonce":1484264540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6829","nonce":1484264540}]},"public_addr":"172.21.15.195:6825/1484264540","cluster_addr":"172.21.15.195:6827/1484264540","heartbeat_back_addr":"172.21.15.195:6831/1484264540","heartbeat_front_addr":"172.21.15.195:6829/1484264540","state":["exists","up"]},{"osd":6,"uuid":"38b5f165-bf91-467e-a615-4b4784337689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6811","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6813","nonce":2638471412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6817","nonce":2638471412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6823","nonce":2638471412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6821","nonce":2638471412}]},"public_addr":"172.21.15.195:6813/2638471412","cluster_addr":"172.21.15.195:6817/2638471412","heartbeat_back_addr":"172.21.15.195:6823/2638471412","heartbeat_front_addr":"172.21.15.195:6821/2638471412","state":["exists","up"]},{"osd":7,"uuid":"e5b3c596-494b-4df3-9437-bc2d52168ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6809","nonce":768189614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6812","nonce":768189614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6820","nonce":768189614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6816","nonce":768189614}]},"public_addr":"172.21.15.195:6809/768189614","cluster_addr":"172.21.15.195:6812/768189614","heartbeat_back_addr":"172.21.15.195:6820/768189614","heartbeat_front_addr":"172.21.15.195:6816/768189614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.209077+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.086601+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.221568+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.238524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.059477+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.148422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.079867+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.064916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T05:39:53.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.452+0000 7f660bc34640 1 -- 172.21.15.107:0/382262542 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f65e80668d0 msgr2=0x7f65e8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:53.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.452+0000 7f660bc34640 1 --2- 172.21.15.107:0/382262542 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f65e80668d0 0x7f65e8068d90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f66000b2000 tx=0x7f66000aa070 comp rx=0 tx=0).stop 2023-12-04T05:39:53.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.452+0000 7f660bc34640 1 -- 172.21.15.107:0/382262542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66040ca2d0 msgr2=0x7f66040cc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:53.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.452+0000 7f660bc34640 1 --2- 172.21.15.107:0/382262542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66040ca2d0 0x7f66040cc630 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f65f400c480 tx=0x7f65f400c940 comp rx=0 tx=0).stop 2023-12-04T05:39:53.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.452+0000 7f660bc34640 1 -- 172.21.15.107:0/382262542 shutdown_connections 2023-12-04T05:39:53.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.452+0000 7f660bc34640 1 --2- 172.21.15.107:0/382262542 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f65e80668d0 0x7f65e8068d90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.452+0000 7f660bc34640 1 --2- 172.21.15.107:0/382262542 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6604144440 0x7f66041f11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.452+0000 7f660bc34640 1 --2- 172.21.15.107:0/382262542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66040ca2d0 0x7f66040cc630 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.452+0000 7f660bc34640 1 --2- 172.21.15.107:0/382262542 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6604072ab0 0x7f6604150e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.452+0000 7f660bc34640 1 -- 172.21.15.107:0/382262542 >> 172.21.15.107:0/382262542 conn(0x7f66040663a0 msgr2=0x7f6604158670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:53.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.452+0000 7f660bc34640 1 -- 172.21.15.107:0/382262542 shutdown_connections 2023-12-04T05:39:53.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.452+0000 7f660bc34640 1 -- 172.21.15.107:0/382262542 wait complete. 2023-12-04T05:39:53.472 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-04T05:39:47.567702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-04T05:39:53.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-04T05:39:53.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a14a03640 1 Processor -- start 2023-12-04T05:39:53.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a14a03640 1 -- start start 2023-12-04T05:39:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a14a03640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9a10072500 0x7f9a10072990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a14a03640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a100c8a10 con 0x7f9a101541f0 2023-12-04T05:39:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a14a03640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a10072ed0 con 0x7f9a10153e10 2023-12-04T05:39:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a14a03640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1015af20 con 0x7f9a10072500 2023-12-04T05:39:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0dd9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9a101541f0 0x7f9a100ca010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53354/0 (socket says 172.21.15.107:53354) 2023-12-04T05:39:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0dd9b640 1 -- 172.21.15.107:0/2206187820 learned_addr learned my addr 172.21.15.107:0/2206187820 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:39:53.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0d59a640 1 -- 172.21.15.107:0/2206187820 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575247405 0 0) 0x7f9a100c8a10 con 0x7f9a101541f0 2023-12-04T05:39:53.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0d59a640 1 -- 172.21.15.107:0/2206187820 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f8003530 con 0x7f9a101541f0 2023-12-04T05:39:53.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0ed9d640 1 --2- 172.21.15.107:0/2206187820 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9a10072500 0x7f9a10072990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0d59a640 1 -- 172.21.15.107:0/2206187820 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1380927009 0 0) 0x7f99f8003530 con 0x7f9a101541f0 2023-12-04T05:39:53.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0d59a640 1 -- 172.21.15.107:0/2206187820 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9a10072500 msgr2=0x7f9a10072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:53.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0d59a640 1 --2- 172.21.15.107:0/2206187820 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9a10072500 0x7f9a10072990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0d59a640 1 -- 172.21.15.107:0/2206187820 >> v1:172.21.15.195:6789/0 conn(0x7f9a10153e10 legacy=0x7f9a1015dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:53.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0d59a640 1 -- 172.21.15.107:0/2206187820 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1006d2e0 con 0x7f9a101541f0 2023-12-04T05:39:53.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0d59a640 1 -- 172.21.15.107:0/2206187820 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f9a00001680 con 0x7f9a101541f0 2023-12-04T05:39:53.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0d59a640 1 -- 172.21.15.107:0/2206187820 >> v1:172.21.15.107:6789/0 conn(0x7f9a101541f0 legacy=0x7f9a100ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:53.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0d59a640 1 --2- 172.21.15.107:0/2206187820 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9a10072500 0x7f99f80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0d59a640 1 --2- 172.21.15.107:0/2206187820 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f99f8003f30 0x7f99f80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0d59a640 1 --2- 172.21.15.107:0/2206187820 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99f8004d10 0x7f99f8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0d59a640 1 -- 172.21.15.107:0/2206187820 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1015af20 con 0x7f99f8004d10 2023-12-04T05:39:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0d59a640 1 -- 172.21.15.107:0/2206187820 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f800bb00 con 0x7f99f8003f30 2023-12-04T05:39:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0d59a640 1 -- 172.21.15.107:0/2206187820 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f800bcc0 con 0x7f9a10072500 2023-12-04T05:39:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0ed9d640 1 --2- 172.21.15.107:0/2206187820 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99f8004d10 0x7f99f8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0e59c640 1 --2- 172.21.15.107:0/2206187820 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9a10072500 0x7f99f80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0dd9b640 1 --2- 172.21.15.107:0/2206187820 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f99f8003f30 0x7f99f80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0ed9d640 1 -- 172.21.15.107:0/2206187820 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9a10072500 msgr2=0x7f99f80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0ed9d640 1 --2- 172.21.15.107:0/2206187820 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9a10072500 0x7f99f80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0ed9d640 1 -- 172.21.15.107:0/2206187820 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f99f8003f30 msgr2=0x7f99f80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0ed9d640 1 --2- 172.21.15.107:0/2206187820 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f99f8003f30 0x7f99f80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0ed9d640 1 -- 172.21.15.107:0/2206187820 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f8003530 con 0x7f99f8004d10 2023-12-04T05:39:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.580+0000 7f9a0e59c640 1 --2- 172.21.15.107:0/2206187820 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9a10072500 0x7f99f80039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:39:53.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a0ed9d640 1 --2- 172.21.15.107:0/2206187820 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99f8004d10 0x7f99f8005190 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9a08009fc0 tx=0x7f9a0801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6486c91287dc4635 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:53.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a0d59a640 1 -- 172.21.15.107:0/2206187820 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a080a5c70 con 0x7f99f8004d10 2023-12-04T05:39:53.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a0d59a640 1 -- 172.21.15.107:0/2206187820 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a0800f040 con 0x7f99f8004d10 2023-12-04T05:39:53.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a0d59a640 1 -- 172.21.15.107:0/2206187820 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a080ad440 con 0x7f99f8004d10 2023-12-04T05:39:53.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a14a03640 1 -- 172.21.15.107:0/2206187820 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99f8004d10 msgr2=0x7f99f8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:53.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a14a03640 1 --2- 172.21.15.107:0/2206187820 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99f8004d10 0x7f99f8005190 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9a08009fc0 tx=0x7f9a0801cc60 comp rx=0 tx=0).stop 2023-12-04T05:39:53.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a0ed9d640 1 -- 172.21.15.107:0/2206187820 reap_dead start 2023-12-04T05:39:53.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a14a03640 1 -- 172.21.15.107:0/2206187820 shutdown_connections 2023-12-04T05:39:53.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a14a03640 1 -- 172.21.15.107:0/2206187820 >> 172.21.15.107:0/2206187820 conn(0x7f9a100663a0 msgr2=0x7f9a100667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:53.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a14a03640 1 -- 172.21.15.107:0/2206187820 shutdown_connections 2023-12-04T05:39:53.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a14a03640 1 -- 172.21.15.107:0/2206187820 wait complete. 2023-12-04T05:39:53.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a14a03640 1 Processor -- start 2023-12-04T05:39:53.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a14a03640 1 -- start start 2023-12-04T05:39:53.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a14a03640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f99f8004d10 0x7f9a101f0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a14a03640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a101595e0 0x7f9a10159a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a14a03640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a101572a0 0x7f9a10157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a14a03640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a10154870 con 0x7f9a101595e0 2023-12-04T05:39:53.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a14a03640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a10157c60 con 0x7f9a101572a0 2023-12-04T05:39:53.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a14a03640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a101f0b60 con 0x7f99f8004d10 2023-12-04T05:39:53.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a0dd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a101595e0 0x7f9a10159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a0dd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a101595e0 0x7f9a10159a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33862/0 (socket says 172.21.15.107:33862) 2023-12-04T05:39:53.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a0dd9b640 1 -- 172.21.15.107:0/2142769391 learned_addr learned my addr 172.21.15.107:0/2142769391 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:39:53.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a0ed9d640 1 --2- 172.21.15.107:0/2142769391 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a101572a0 0x7f9a10157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a0dd9b640 1 -- 172.21.15.107:0/2142769391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f99f8004d10 msgr2=0x7f9a101f0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:53.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a0e59c640 1 --2- 172.21.15.107:0/2142769391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f99f8004d10 0x7f9a101f0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a0dd9b640 1 --2- 172.21.15.107:0/2142769391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f99f8004d10 0x7f9a101f0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a0dd9b640 1 -- 172.21.15.107:0/2142769391 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a101572a0 msgr2=0x7f9a10157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:53.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a0dd9b640 1 --2- 172.21.15.107:0/2142769391 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a101572a0 0x7f9a10157720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a0dd9b640 1 -- 172.21.15.107:0/2142769391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a10159150 con 0x7f9a101595e0 2023-12-04T05:39:53.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.584+0000 7f9a0ed9d640 1 --2- 172.21.15.107:0/2142769391 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a101572a0 0x7f9a10157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:39:53.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.588+0000 7f9a0dd9b640 1 --2- 172.21.15.107:0/2142769391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a101595e0 0x7f9a10159a20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9a0000e7f0 tx=0x7f9a0000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:53.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.588+0000 7f99feffd640 1 -- 172.21.15.107:0/2142769391 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0000c8d0 con 0x7f9a101595e0 2023-12-04T05:39:53.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.588+0000 7f99feffd640 1 -- 172.21.15.107:0/2142769391 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a0000ca30 con 0x7f9a101595e0 2023-12-04T05:39:53.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.588+0000 7f9a14a03640 1 -- 172.21.15.107:0/2142769391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a1014b2d0 con 0x7f9a101595e0 2023-12-04T05:39:53.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.588+0000 7f99feffd640 1 -- 172.21.15.107:0/2142769391 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0000cde0 con 0x7f9a101595e0 2023-12-04T05:39:53.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.588+0000 7f9a14a03640 1 -- 172.21.15.107:0/2142769391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a1014aff0 con 0x7f9a101595e0 2023-12-04T05:39:53.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.588+0000 7f9a14a03640 1 -- 172.21.15.107:0/2142769391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a101592e0 con 0x7f9a101595e0 2023-12-04T05:39:53.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.588+0000 7f99feffd640 1 -- 172.21.15.107:0/2142769391 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9a0001d460 con 0x7f9a101595e0 2023-12-04T05:39:53.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.588+0000 7f99feffd640 1 --2- 172.21.15.107:0/2142769391 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f99f00668d0 0x7f99f0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.588+0000 7f99feffd640 1 -- 172.21.15.107:0/2142769391 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9a00084220 con 0x7f9a101595e0 2023-12-04T05:39:53.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.588+0000 7f9a0e59c640 1 --2- 172.21.15.107:0/2142769391 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f99f00668d0 0x7f99f0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.588+0000 7f9a0e59c640 1 --2- 172.21.15.107:0/2142769391 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f99f00668d0 0x7f99f0068d90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9a04001680 tx=0x7f9a04009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:53.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.592+0000 7f99feffd640 1 -- 172.21.15.107:0/2142769391 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9a00051720 con 0x7f9a101595e0 2023-12-04T05:39:53.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.748+0000 7f9a14a03640 1 -- 172.21.15.107:0/2142769391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9a10067ec0 con 0x7f9a101595e0 2023-12-04T05:39:53.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.748+0000 7f99feffd640 1 -- 172.21.15.107:0/2142769391 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9a00054d40 con 0x7f9a101595e0 2023-12-04T05:39:53.753 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-12-04T05:39:53.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.752+0000 7f9a14a03640 1 -- 172.21.15.107:0/2142769391 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f99f00668d0 msgr2=0x7f99f0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:53.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.752+0000 7f9a14a03640 1 --2- 172.21.15.107:0/2142769391 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f99f00668d0 0x7f99f0068d90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9a04001680 tx=0x7f9a04009040 comp rx=0 tx=0).stop 2023-12-04T05:39:53.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.752+0000 7f9a14a03640 1 -- 172.21.15.107:0/2142769391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a101595e0 msgr2=0x7f9a10159a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:53.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.752+0000 7f9a14a03640 1 --2- 172.21.15.107:0/2142769391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a101595e0 0x7f9a10159a20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9a0000e7f0 tx=0x7f9a0000ecb0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.752+0000 7f9a14a03640 1 -- 172.21.15.107:0/2142769391 shutdown_connections 2023-12-04T05:39:53.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.752+0000 7f9a14a03640 1 --2- 172.21.15.107:0/2142769391 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f99f00668d0 0x7f99f0068d90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.752+0000 7f9a14a03640 1 --2- 172.21.15.107:0/2142769391 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a101572a0 0x7f9a10157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.752+0000 7f9a14a03640 1 --2- 172.21.15.107:0/2142769391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a101595e0 0x7f9a10159a20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.752+0000 7f9a14a03640 1 --2- 172.21.15.107:0/2142769391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f99f8004d10 0x7f9a101f0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.752+0000 7f9a14a03640 1 -- 172.21.15.107:0/2142769391 >> 172.21.15.107:0/2142769391 conn(0x7f9a100663a0 msgr2=0x7f9a100667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:53.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.752+0000 7f9a14a03640 1 -- 172.21.15.107:0/2142769391 shutdown_connections 2023-12-04T05:39:53.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.752+0000 7f9a14a03640 1 -- 172.21.15.107:0/2142769391 wait complete. 2023-12-04T05:39:53.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:39:53.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85b4c25640 1 Processor -- start 2023-12-04T05:39:53.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85b4c25640 1 -- start start 2023-12-04T05:39:53.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85b4c25640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f85b00c46a0 0x7f85b00c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85b4c25640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f85b0157180 0x7f85b00c7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85b4c25640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b006e660 con 0x7f85b00c4a80 2023-12-04T05:39:53.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85b4c25640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85b0156490 con 0x7f85b00c46a0 2023-12-04T05:39:53.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85b4c25640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b0156cf0 con 0x7f85b0157180 2023-12-04T05:39:53.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85aed9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f85b00c4a80 0x7f85b00c5ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53356/0 (socket says 172.21.15.107:53356) 2023-12-04T05:39:53.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85aed9d640 1 -- 172.21.15.107:0/3669167650 learned_addr learned my addr 172.21.15.107:0/3669167650 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:39:53.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85ae59c640 1 --2- 172.21.15.107:0/3669167650 >> v2:172.21.15.195:3300/0 conn(0x7f85b00c46a0 0x7f85b00c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85add9b640 1 --2- 172.21.15.107:0/3669167650 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f85b0157180 0x7f85b00c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85ad59a640 1 -- 172.21.15.107:0/3669167650 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513115967 0 0) 0x7f85a80014a0 con 0x7f85b00c4a80 2023-12-04T05:39:53.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85ad59a640 1 -- 172.21.15.107:0/3669167650 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8598003530 con 0x7f85b00c4a80 2023-12-04T05:39:53.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85ad59a640 1 -- 172.21.15.107:0/3669167650 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1378410839 0 0) 0x7f8598003530 con 0x7f85b00c4a80 2023-12-04T05:39:53.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85ad59a640 1 -- 172.21.15.107:0/3669167650 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f85b0157180 msgr2=0x7f85b00c7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:53.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85ad59a640 1 --2- 172.21.15.107:0/3669167650 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f85b0157180 0x7f85b00c7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85ad59a640 1 -- 172.21.15.107:0/3669167650 >> v2:172.21.15.195:3300/0 conn(0x7f85b00c46a0 msgr2=0x7f85b00c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:53.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85ad59a640 1 --2- 172.21.15.107:0/3669167650 >> v2:172.21.15.195:3300/0 conn(0x7f85b00c46a0 0x7f85b00c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85ad59a640 1 -- 172.21.15.107:0/3669167650 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b006e0a0 con 0x7f85b00c4a80 2023-12-04T05:39:53.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85ad59a640 1 -- 172.21.15.107:0/3669167650 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f85a8001b90 con 0x7f85b00c4a80 2023-12-04T05:39:53.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85ad59a640 1 -- 172.21.15.107:0/3669167650 >> v1:172.21.15.107:6789/0 conn(0x7f85b00c4a80 legacy=0x7f85b00c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:53.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85ad59a640 1 --2- 172.21.15.107:0/3669167650 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f85b0157180 0x7f8598003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85ad59a640 1 --2- 172.21.15.107:0/3669167650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8598003f00 0x7f85980047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85ad59a640 1 --2- 172.21.15.107:0/3669167650 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8598004d10 0x7f8598005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85ad59a640 1 -- 172.21.15.107:0/3669167650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b0156490 con 0x7f8598003f00 2023-12-04T05:39:53.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85ad59a640 1 -- 172.21.15.107:0/3669167650 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b0156cf0 con 0x7f8598004d10 2023-12-04T05:39:53.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85ad59a640 1 -- 172.21.15.107:0/3669167650 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f859800b990 con 0x7f85b0157180 2023-12-04T05:39:53.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85add9b640 1 --2- 172.21.15.107:0/3669167650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8598003f00 0x7f85980047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85ae59c640 1 --2- 172.21.15.107:0/3669167650 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f85b0157180 0x7f8598003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85aed9d640 1 --2- 172.21.15.107:0/3669167650 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8598004d10 0x7f8598005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85add9b640 1 -- 172.21.15.107:0/3669167650 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f85b0157180 msgr2=0x7f8598003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:53.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85add9b640 1 --2- 172.21.15.107:0/3669167650 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f85b0157180 0x7f8598003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85add9b640 1 -- 172.21.15.107:0/3669167650 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8598004d10 msgr2=0x7f8598005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:53.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85add9b640 1 --2- 172.21.15.107:0/3669167650 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8598004d10 0x7f8598005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85add9b640 1 -- 172.21.15.107:0/3669167650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8598003530 con 0x7f8598003f00 2023-12-04T05:39:53.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.876+0000 7f85aed9d640 1 --2- 172.21.15.107:0/3669167650 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8598004d10 0x7f8598005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:39:53.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85add9b640 1 --2- 172.21.15.107:0/3669167650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8598003f00 0x7f85980047d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f85a4009ae0 tx=0x7f85a401ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cfe667d6ffe34f2 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:53.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85ad59a640 1 -- 172.21.15.107:0/3669167650 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a40a79d0 con 0x7f8598003f00 2023-12-04T05:39:53.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85ad59a640 1 -- 172.21.15.107:0/3669167650 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a40a7b30 con 0x7f8598003f00 2023-12-04T05:39:53.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85ad59a640 1 -- 172.21.15.107:0/3669167650 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a40af440 con 0x7f8598003f00 2023-12-04T05:39:53.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85b4c25640 1 -- 172.21.15.107:0/3669167650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8598003f00 msgr2=0x7f85980047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:53.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85b4c25640 1 --2- 172.21.15.107:0/3669167650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8598003f00 0x7f85980047d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f85a4009ae0 tx=0x7f85a401ec80 comp rx=0 tx=0).stop 2023-12-04T05:39:53.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85aed9d640 1 -- 172.21.15.107:0/3669167650 reap_dead start 2023-12-04T05:39:53.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85b4c25640 1 -- 172.21.15.107:0/3669167650 shutdown_connections 2023-12-04T05:39:53.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85b4c25640 1 -- 172.21.15.107:0/3669167650 >> 172.21.15.107:0/3669167650 conn(0x7f85b00c7ff0 msgr2=0x7f85b00c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:53.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85b4c25640 1 -- 172.21.15.107:0/3669167650 shutdown_connections 2023-12-04T05:39:53.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85b4c25640 1 -- 172.21.15.107:0/3669167650 wait complete. 2023-12-04T05:39:53.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85b4c25640 1 Processor -- start 2023-12-04T05:39:53.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85b4c25640 1 -- start start 2023-12-04T05:39:53.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85b4c25640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8598003f00 0x7f85b01ef2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85b4c25640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f85b0152440 0x7f85b00709a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85b4c25640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f85b0151ea0 0x7f85b014aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85b4c25640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b0154520 con 0x7f8598003f00 2023-12-04T05:39:53.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85b4c25640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b01ef810 con 0x7f85b0151ea0 2023-12-04T05:39:53.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85b4c25640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b014eb70 con 0x7f85b0152440 2023-12-04T05:39:53.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85ae59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8598003f00 0x7f85b01ef2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85ae59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8598003f00 0x7f85b01ef2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33874/0 (socket says 172.21.15.107:33874) 2023-12-04T05:39:53.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85ae59c640 1 -- 172.21.15.107:0/3982792460 learned_addr learned my addr 172.21.15.107:0/3982792460 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:39:53.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85aed9d640 1 --2- 172.21.15.107:0/3982792460 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f85b0151ea0 0x7f85b014aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85add9b640 1 --2- 172.21.15.107:0/3982792460 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f85b0152440 0x7f85b00709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.880+0000 7f85ae59c640 1 -- 172.21.15.107:0/3982792460 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f85b0152440 msgr2=0x7f85b00709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:53.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.884+0000 7f85ae59c640 1 --2- 172.21.15.107:0/3982792460 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f85b0152440 0x7f85b00709a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.884+0000 7f85ae59c640 1 -- 172.21.15.107:0/3982792460 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f85b0151ea0 msgr2=0x7f85b014aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:53.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.884+0000 7f85ae59c640 1 --2- 172.21.15.107:0/3982792460 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f85b0151ea0 0x7f85b014aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:53.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.884+0000 7f85ae59c640 1 -- 172.21.15.107:0/3982792460 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b014ee30 con 0x7f8598003f00 2023-12-04T05:39:53.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.884+0000 7f85add9b640 1 --2- 172.21.15.107:0/3982792460 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f85b0152440 0x7f85b00709a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:39:53.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.884+0000 7f85ae59c640 1 --2- 172.21.15.107:0/3982792460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8598003f00 0x7f85b01ef2d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f85a0011660 tx=0x7f85a0011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:53.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.884+0000 7f859effd640 1 -- 172.21.15.107:0/3982792460 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a000f9c0 con 0x7f8598003f00 2023-12-04T05:39:53.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.884+0000 7f859effd640 1 -- 172.21.15.107:0/3982792460 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a000fb20 con 0x7f8598003f00 2023-12-04T05:39:53.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.884+0000 7f85b4c25640 1 -- 172.21.15.107:0/3982792460 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b0150eb0 con 0x7f8598003f00 2023-12-04T05:39:53.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.884+0000 7f859effd640 1 -- 172.21.15.107:0/3982792460 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a000fe90 con 0x7f8598003f00 2023-12-04T05:39:53.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.884+0000 7f85b4c25640 1 -- 172.21.15.107:0/3982792460 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b0070ee0 con 0x7f8598003f00 2023-12-04T05:39:53.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.884+0000 7f85b4c25640 1 -- 172.21.15.107:0/3982792460 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b0067870 con 0x7f8598003f00 2023-12-04T05:39:53.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.884+0000 7f859effd640 1 -- 172.21.15.107:0/3982792460 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f85a0015460 con 0x7f8598003f00 2023-12-04T05:39:53.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.884+0000 7f859effd640 1 --2- 172.21.15.107:0/3982792460 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f85900668d0 0x7f8590068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:53.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.884+0000 7f859effd640 1 -- 172.21.15.107:0/3982792460 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f85a0084670 con 0x7f8598003f00 2023-12-04T05:39:53.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.884+0000 7f85add9b640 1 --2- 172.21.15.107:0/3982792460 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f85900668d0 0x7f8590068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:53.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.884+0000 7f85add9b640 1 --2- 172.21.15.107:0/3982792460 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f85900668d0 0x7f8590068d90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f85a40b1000 tx=0x7f85a40046d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:53.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:53.888+0000 7f859effd640 1 -- 172.21.15.107:0/3982792460 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f85a0051c00 con 0x7f8598003f00 2023-12-04T05:39:54.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:54.060+0000 7f85b4c25640 1 -- 172.21.15.107:0/3982792460 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85b014d020 con 0x7f8598003f00 2023-12-04T05:39:54.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:54.060+0000 7f859effd640 1 -- 172.21.15.107:0/3982792460 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1327 (secure 0 0 0) 0x7f85a0022540 con 0x7f8598003f00 2023-12-04T05:39:54.066 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:54.066 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-12-04T05:39:54.066 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 2 2023-12-04T05:39:54.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:54.064+0000 7f85b4c25640 1 -- 172.21.15.107:0/3982792460 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f85900668d0 msgr2=0x7f8590068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:54.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:54.064+0000 7f85b4c25640 1 --2- 172.21.15.107:0/3982792460 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f85900668d0 0x7f8590068d90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f85a40b1000 tx=0x7f85a40046d0 comp rx=0 tx=0).stop 2023-12-04T05:39:54.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:54.064+0000 7f85b4c25640 1 -- 172.21.15.107:0/3982792460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8598003f00 msgr2=0x7f85b01ef2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:54.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:54.064+0000 7f85b4c25640 1 --2- 172.21.15.107:0/3982792460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8598003f00 0x7f85b01ef2d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f85a0011660 tx=0x7f85a0011b20 comp rx=0 tx=0).stop 2023-12-04T05:39:54.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:54.064+0000 7f85b4c25640 1 -- 172.21.15.107:0/3982792460 shutdown_connections 2023-12-04T05:39:54.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:54.064+0000 7f85b4c25640 1 --2- 172.21.15.107:0/3982792460 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f85900668d0 0x7f8590068d90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:54.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:54.064+0000 7f85b4c25640 1 --2- 172.21.15.107:0/3982792460 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f85b0151ea0 0x7f85b014aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:54.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:54.064+0000 7f85b4c25640 1 --2- 172.21.15.107:0/3982792460 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f85b0152440 0x7f85b00709a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:54.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:54.064+0000 7f85b4c25640 1 --2- 172.21.15.107:0/3982792460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8598003f00 0x7f85b01ef2d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:54.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:54.064+0000 7f85b4c25640 1 -- 172.21.15.107:0/3982792460 >> 172.21.15.107:0/3982792460 conn(0x7f85b00c7ff0 msgr2=0x7f85b01e1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:54.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:54.064+0000 7f85b4c25640 1 -- 172.21.15.107:0/3982792460 shutdown_connections 2023-12-04T05:39:54.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:54.064+0000 7f85b4c25640 1 -- 172.21.15.107:0/3982792460 wait complete. 2023-12-04T05:39:56.083 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:39:56.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.212+0000 7f3bb7256640 1 Processor -- start 2023-12-04T05:39:56.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.212+0000 7f3bb7256640 1 -- start start 2023-12-04T05:39:56.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.212+0000 7f3bb7256640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3bb0151180 0x7f3bb00c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:56.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.212+0000 7f3bb7256640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3bb0070380 0x7f3bb0070780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:56.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.212+0000 7f3bb7256640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb00700c0 con 0x7f3bb00714f0 2023-12-04T05:39:56.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.212+0000 7f3bb7256640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bb0156df0 con 0x7f3bb0151180 2023-12-04T05:39:56.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.212+0000 7f3bb7256640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bb01570d0 con 0x7f3bb0070380 2023-12-04T05:39:56.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.212+0000 7f3bb4ff2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3bb0151180 0x7f3bb00c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:56.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.212+0000 7f3bb4ff2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3bb0151180 0x7f3bb00c79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33876/0 (socket says 172.21.15.107:33876) 2023-12-04T05:39:56.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.212+0000 7f3bb4ff2640 1 -- 172.21.15.107:0/3582403048 learned_addr learned my addr 172.21.15.107:0/3582403048 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:39:56.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.212+0000 7f3bb57f3640 1 --2- 172.21.15.107:0/3582403048 >> v2:172.21.15.195:3300/0 conn(0x7f3bb0070380 0x7f3bb0070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:56.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.212+0000 7f3bb4ff2640 1 -- 172.21.15.107:0/3582403048 >> v2:172.21.15.195:3300/0 conn(0x7f3bb0070380 msgr2=0x7f3bb0070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:56.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.212+0000 7f3bb4ff2640 1 --2- 172.21.15.107:0/3582403048 >> v2:172.21.15.195:3300/0 conn(0x7f3bb0070380 0x7f3bb0070780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.212+0000 7f3bb4ff2640 1 -- 172.21.15.107:0/3582403048 >> v1:172.21.15.195:6789/0 conn(0x7f3bb00714f0 legacy=0x7f3bb0071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:56.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.212+0000 7f3bb4ff2640 1 -- 172.21.15.107:0/3582403048 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb01573b0 con 0x7f3bb0151180 2023-12-04T05:39:56.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb4ff2640 1 --2- 172.21.15.107:0/3582403048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb0151180 0x7f3bb00c79e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3bb006f680 tx=0x7f3b9801ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=702bb1a6c88ee01 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:56.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3baf7fe640 1 -- 172.21.15.107:0/3582403048 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b980a89b0 con 0x7f3bb0151180 2023-12-04T05:39:56.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3baf7fe640 1 -- 172.21.15.107:0/3582403048 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b980a8b10 con 0x7f3bb0151180 2023-12-04T05:39:56.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3baf7fe640 1 -- 172.21.15.107:0/3582403048 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b980a8ea0 con 0x7f3bb0151180 2023-12-04T05:39:56.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb7256640 1 -- 172.21.15.107:0/3582403048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb0151180 msgr2=0x7f3bb00c79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:56.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb7256640 1 --2- 172.21.15.107:0/3582403048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb0151180 0x7f3bb00c79e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3bb006f680 tx=0x7f3b9801ecb0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb7256640 1 -- 172.21.15.107:0/3582403048 shutdown_connections 2023-12-04T05:39:56.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb7256640 1 --2- 172.21.15.107:0/3582403048 >> v2:172.21.15.195:3300/0 conn(0x7f3bb0070380 0x7f3bb0070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb7256640 1 --2- 172.21.15.107:0/3582403048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb0151180 0x7f3bb00c79e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb7256640 1 -- 172.21.15.107:0/3582403048 >> 172.21.15.107:0/3582403048 conn(0x7f3bb00c7ff0 msgr2=0x7f3bb00c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:56.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb7256640 1 -- 172.21.15.107:0/3582403048 shutdown_connections 2023-12-04T05:39:56.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb7256640 1 -- 172.21.15.107:0/3582403048 wait complete. 2023-12-04T05:39:56.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb7256640 1 Processor -- start 2023-12-04T05:39:56.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb7256640 1 -- start start 2023-12-04T05:39:56.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb7256640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3bb0070380 0x7f3bb01e3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:56.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb7256640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3bb00714f0 0x7f3bb01f0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:56.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb7256640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb0151180 0x7f3bb01f1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:56.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb7256640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb0157e30 con 0x7f3bb0151180 2023-12-04T05:39:56.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb7256640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb0158b20 con 0x7f3bb0070380 2023-12-04T05:39:56.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb7256640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb0158690 con 0x7f3bb00714f0 2023-12-04T05:39:56.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb57f3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb0151180 0x7f3bb01f1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:56.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb57f3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb0151180 0x7f3bb01f1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33878/0 (socket says 172.21.15.107:33878) 2023-12-04T05:39:56.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb57f3640 1 -- 172.21.15.107:0/1729365243 learned_addr learned my addr 172.21.15.107:0/1729365243 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:39:56.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3baffff640 1 --2- 172.21.15.107:0/1729365243 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3bb00714f0 0x7f3bb01f0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:56.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb4ff2640 1 --2- 172.21.15.107:0/1729365243 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3bb0070380 0x7f3bb01e3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:56.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb57f3640 1 -- 172.21.15.107:0/1729365243 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3bb00714f0 msgr2=0x7f3bb01f0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:56.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb57f3640 1 --2- 172.21.15.107:0/1729365243 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3bb00714f0 0x7f3bb01f0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb57f3640 1 -- 172.21.15.107:0/1729365243 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3bb0070380 msgr2=0x7f3bb01e3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:56.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb57f3640 1 --2- 172.21.15.107:0/1729365243 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3bb0070380 0x7f3bb01e3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.216+0000 7f3bb57f3640 1 -- 172.21.15.107:0/1729365243 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b980094d0 con 0x7f3bb0151180 2023-12-04T05:39:56.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.220+0000 7f3bb57f3640 1 --2- 172.21.15.107:0/1729365243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb0151180 0x7f3bb01f1280 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8007e40 tx=0x7f3ba800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:56.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.220+0000 7f3bad7fa640 1 -- 172.21.15.107:0/1729365243 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba8014620 con 0x7f3bb0151180 2023-12-04T05:39:56.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.220+0000 7f3bad7fa640 1 -- 172.21.15.107:0/1729365243 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba800ee90 con 0x7f3bb0151180 2023-12-04T05:39:56.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.220+0000 7f3bb7256640 1 -- 172.21.15.107:0/1729365243 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb00cae60 con 0x7f3bb0151180 2023-12-04T05:39:56.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.220+0000 7f3bad7fa640 1 -- 172.21.15.107:0/1729365243 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba80149e0 con 0x7f3bb0151180 2023-12-04T05:39:56.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.220+0000 7f3bb7256640 1 -- 172.21.15.107:0/1729365243 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb0158040 con 0x7f3bb0151180 2023-12-04T05:39:56.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.220+0000 7f3bb7256640 1 -- 172.21.15.107:0/1729365243 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb0158840 con 0x7f3bb0151180 2023-12-04T05:39:56.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.220+0000 7f3bad7fa640 1 -- 172.21.15.107:0/1729365243 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f3ba8014b40 con 0x7f3bb0151180 2023-12-04T05:39:56.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.220+0000 7f3bad7fa640 1 --2- 172.21.15.107:0/1729365243 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3b90066900 0x7f3b90068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:56.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.220+0000 7f3bad7fa640 1 -- 172.21.15.107:0/1729365243 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3ba8084a60 con 0x7f3bb0151180 2023-12-04T05:39:56.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.220+0000 7f3bb4ff2640 1 --2- 172.21.15.107:0/1729365243 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3b90066900 0x7f3b90068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:56.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.220+0000 7f3bb4ff2640 1 --2- 172.21.15.107:0/1729365243 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3b90066900 0x7f3b90068dc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3b980b2000 tx=0x7f3b98008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:56.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.224+0000 7f3bad7fa640 1 -- 172.21.15.107:0/1729365243 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3bb0158840 con 0x7f3bb0151180 2023-12-04T05:39:56.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.408+0000 7f3bb7256640 1 -- 172.21.15.107:0/1729365243 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bb0158800 con 0x7f3bb0151180 2023-12-04T05:39:56.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.412+0000 7f3bad7fa640 1 -- 172.21.15.107:0/1729365243 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5286 (secure 0 0 0) 0x7f3ba8051ff0 con 0x7f3bb0151180 2023-12-04T05:39:56.412 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:56.412 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6836/3769196693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":3769196693},{"type":"v1","addr":"172.21.15.195:6836","nonce":3769196693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4288,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-12-04T05:39:56.412 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 3 2023-12-04T05:39:56.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.412+0000 7f3bb7256640 1 -- 172.21.15.107:0/1729365243 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3b90066900 msgr2=0x7f3b90068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:56.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.412+0000 7f3bb7256640 1 --2- 172.21.15.107:0/1729365243 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3b90066900 0x7f3b90068dc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3b980b2000 tx=0x7f3b98008040 comp rx=0 tx=0).stop 2023-12-04T05:39:56.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.412+0000 7f3bb7256640 1 -- 172.21.15.107:0/1729365243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb0151180 msgr2=0x7f3bb01f1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:56.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.412+0000 7f3bb7256640 1 --2- 172.21.15.107:0/1729365243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb0151180 0x7f3bb01f1280 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8007e40 tx=0x7f3ba800e4b0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.412+0000 7f3bb7256640 1 -- 172.21.15.107:0/1729365243 shutdown_connections 2023-12-04T05:39:56.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.412+0000 7f3bb7256640 1 --2- 172.21.15.107:0/1729365243 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3b90066900 0x7f3b90068dc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.412+0000 7f3bb7256640 1 --2- 172.21.15.107:0/1729365243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb0151180 0x7f3bb01f1280 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.412+0000 7f3bb7256640 1 --2- 172.21.15.107:0/1729365243 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3bb00714f0 0x7f3bb01f0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.412+0000 7f3bb7256640 1 --2- 172.21.15.107:0/1729365243 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3bb0070380 0x7f3bb01e3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.412+0000 7f3bb7256640 1 -- 172.21.15.107:0/1729365243 >> 172.21.15.107:0/1729365243 conn(0x7f3bb00c7ff0 msgr2=0x7f3bb0070780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:56.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.412+0000 7f3bb7256640 1 -- 172.21.15.107:0/1729365243 shutdown_connections 2023-12-04T05:39:56.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.412+0000 7f3bb7256640 1 -- 172.21.15.107:0/1729365243 wait complete. 2023-12-04T05:39:56.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T05:39:56.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.536+0000 7f763ee05640 1 Processor -- start 2023-12-04T05:39:56.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.536+0000 7f763ee05640 1 -- start start 2023-12-04T05:39:56.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.540+0000 7f763ee05640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7638072110 0x7f7638158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:56.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.540+0000 7f763ee05640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f76380724f0 0x7f76380c7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:56.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.540+0000 7f763ee05640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7638158da0 con 0x7f76380c56f0 2023-12-04T05:39:56.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.540+0000 7f763ee05640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76381583d0 con 0x7f76380724f0 2023-12-04T05:39:56.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.540+0000 7f763ee05640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76380c6170 con 0x7f7638072110 2023-12-04T05:39:56.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.540+0000 7f763cba1640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7638072110 0x7f7638158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:56.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.540+0000 7f762ffff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f76380724f0 0x7f76380c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:56.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.540+0000 7f763cba1640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7638072110 0x7f7638158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.107:36344/0 (socket says 172.21.15.107:36344) 2023-12-04T05:39:56.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.540+0000 7f763cba1640 1 -- 172.21.15.107:0/600084862 learned_addr learned my addr 172.21.15.107:0/600084862 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:39:56.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.540+0000 7f763d3a2640 1 --1- 172.21.15.107:0/600084862 >> v1:172.21.15.195:6789/0 conn(0x7f76380c56f0 0x7f7638140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:58620/0 (socket says 172.21.15.107:58620) 2023-12-04T05:39:56.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.540+0000 7f763cba1640 1 -- 172.21.15.107:0/600084862 >> v2:172.21.15.195:3300/0 conn(0x7f76380724f0 msgr2=0x7f76380c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:56.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.540+0000 7f763cba1640 1 --2- 172.21.15.107:0/600084862 >> v2:172.21.15.195:3300/0 conn(0x7f76380724f0 0x7f76380c7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.540+0000 7f763cba1640 1 -- 172.21.15.107:0/600084862 >> v1:172.21.15.195:6789/0 conn(0x7f76380c56f0 legacy=0x7f7638140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:56.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.540+0000 7f763cba1640 1 -- 172.21.15.107:0/600084862 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7638157da0 con 0x7f7638072110 2023-12-04T05:39:56.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.540+0000 7f762ffff640 1 --2- 172.21.15.107:0/600084862 >> v2:172.21.15.195:3300/0 conn(0x7f76380724f0 0x7f76380c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T05:39:56.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.540+0000 7f763cba1640 1 --2- 172.21.15.107:0/600084862 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7638072110 0x7f7638158860 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7634009870 tx=0x7f763401c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f09fce2754b520d server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:56.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f762f7fe640 1 -- 172.21.15.107:0/600084862 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76340a7070 con 0x7f7638072110 2023-12-04T05:39:56.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f762f7fe640 1 -- 172.21.15.107:0/600084862 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76340ab430 con 0x7f7638072110 2023-12-04T05:39:56.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f762f7fe640 1 -- 172.21.15.107:0/600084862 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76340ab7c0 con 0x7f7638072110 2023-12-04T05:39:56.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763ee05640 1 -- 172.21.15.107:0/600084862 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7638072110 msgr2=0x7f7638158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:56.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763ee05640 1 --2- 172.21.15.107:0/600084862 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7638072110 0x7f7638158860 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7634009870 tx=0x7f763401c540 comp rx=0 tx=0).stop 2023-12-04T05:39:56.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763ee05640 1 -- 172.21.15.107:0/600084862 shutdown_connections 2023-12-04T05:39:56.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763ee05640 1 --2- 172.21.15.107:0/600084862 >> v2:172.21.15.195:3300/0 conn(0x7f76380724f0 0x7f76380c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763ee05640 1 --2- 172.21.15.107:0/600084862 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7638072110 0x7f7638158860 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763ee05640 1 -- 172.21.15.107:0/600084862 >> 172.21.15.107:0/600084862 conn(0x7f76380c7ff0 msgr2=0x7f76380c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:56.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763ee05640 1 -- 172.21.15.107:0/600084862 shutdown_connections 2023-12-04T05:39:56.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763ee05640 1 -- 172.21.15.107:0/600084862 wait complete. 2023-12-04T05:39:56.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763ee05640 1 Processor -- start 2023-12-04T05:39:56.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763ee05640 1 -- start start 2023-12-04T05:39:56.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763ee05640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638072110 0x7f763813d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:56.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763ee05640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f76380724f0 0x7f7638152930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:56.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763ee05640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f76380c56f0 0x7f76381626f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:56.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763ee05640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7638153d60 con 0x7f7638072110 2023-12-04T05:39:56.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763ee05640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76381f3510 con 0x7f76380c56f0 2023-12-04T05:39:56.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763ee05640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76381f2c20 con 0x7f76380724f0 2023-12-04T05:39:56.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763cba1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638072110 0x7f763813d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:56.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763cba1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638072110 0x7f763813d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33880/0 (socket says 172.21.15.107:33880) 2023-12-04T05:39:56.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763cba1640 1 -- 172.21.15.107:0/161208018 learned_addr learned my addr 172.21.15.107:0/161208018 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:39:56.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f762ffff640 1 --2- 172.21.15.107:0/161208018 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f76380724f0 0x7f7638152930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:56.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763d3a2640 1 --2- 172.21.15.107:0/161208018 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f76380c56f0 0x7f76381626f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:56.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763cba1640 1 -- 172.21.15.107:0/161208018 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f76380724f0 msgr2=0x7f7638152930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:56.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763cba1640 1 --2- 172.21.15.107:0/161208018 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f76380724f0 0x7f7638152930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763cba1640 1 -- 172.21.15.107:0/161208018 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f76380c56f0 msgr2=0x7f76381626f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:56.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763cba1640 1 --2- 172.21.15.107:0/161208018 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f76380c56f0 0x7f76381626f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763cba1640 1 -- 172.21.15.107:0/161208018 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7628009420 con 0x7f7638072110 2023-12-04T05:39:56.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763cba1640 1 --2- 172.21.15.107:0/161208018 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638072110 0x7f763813d030 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f76340ad040 tx=0x7f76340192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:56.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f762d7fa640 1 -- 172.21.15.107:0/161208018 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76340b45f0 con 0x7f7638072110 2023-12-04T05:39:56.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f762d7fa640 1 -- 172.21.15.107:0/161208018 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76340b4750 con 0x7f7638072110 2023-12-04T05:39:56.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763ee05640 1 -- 172.21.15.107:0/161208018 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76340094d0 con 0x7f7638072110 2023-12-04T05:39:56.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f762d7fa640 1 -- 172.21.15.107:0/161208018 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76340b4b00 con 0x7f7638072110 2023-12-04T05:39:56.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.544+0000 7f763ee05640 1 -- 172.21.15.107:0/161208018 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76381f2ec0 con 0x7f7638072110 2023-12-04T05:39:56.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.548+0000 7f763ee05640 1 -- 172.21.15.107:0/161208018 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7638067870 con 0x7f7638072110 2023-12-04T05:39:56.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.548+0000 7f762d7fa640 1 -- 172.21.15.107:0/161208018 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f76340b4c60 con 0x7f7638072110 2023-12-04T05:39:56.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.548+0000 7f762d7fa640 1 --2- 172.21.15.107:0/161208018 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7624066900 0x7f7624068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:56.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.548+0000 7f762d7fa640 1 -- 172.21.15.107:0/161208018 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f763411d760 con 0x7f7638072110 2023-12-04T05:39:56.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.548+0000 7f762ffff640 1 --2- 172.21.15.107:0/161208018 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7624066900 0x7f7624068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:56.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.548+0000 7f762ffff640 1 --2- 172.21.15.107:0/161208018 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7624066900 0x7f7624068dc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f76381f0740 tx=0x7f7628008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:56.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.552+0000 7f762d7fa640 1 -- 172.21.15.107:0/161208018 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f76340eacf0 con 0x7f7638072110 2023-12-04T05:39:56.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.704+0000 7f763ee05640 1 -- 172.21.15.107:0/161208018 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76380c67a0 con 0x7f7638072110 2023-12-04T05:39:56.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.708+0000 7f762d7fa640 1 -- 172.21.15.107:0/161208018 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12886 (secure 0 0 0) 0x7f76340bd090 con 0x7f7638072110 2023-12-04T05:39:56.708 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:56.708 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"fsid":"603e856b-31ac-475a-8f35-21ab7faadd48","created":"2023-12-04T05:39:34.921690+0000","modified":"2023-12-04T05:39:50.257517+0000","last_up_change":"2023-12-04T05:39:46.235790+0000","last_in_change":"2023-12-04T05:39:37.587639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T05:39:47.567702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ab7111c-0ae8-4d41-b6db-e3743614322c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6817","nonce":768714279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6821","nonce":768714279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6829","nonce":768714279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6825","nonce":768714279}]},"public_addr":"172.21.15.107:6817/768714279","cluster_addr":"172.21.15.107:6821/768714279","heartbeat_back_addr":"172.21.15.107:6829/768714279","heartbeat_front_addr":"172.21.15.107:6825/768714279","state":["exists","up"]},{"osd":1,"uuid":"baef7a1f-8026-4fab-9e70-48465271034e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6801","nonce":2496355981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6803","nonce":2496355981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6807","nonce":2496355981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6805","nonce":2496355981}]},"public_addr":"172.21.15.107:6801/2496355981","cluster_addr":"172.21.15.107:6803/2496355981","heartbeat_back_addr":"172.21.15.107:6807/2496355981","heartbeat_front_addr":"172.21.15.107:6805/2496355981","state":["exists","up"]},{"osd":2,"uuid":"90e8cf24-7b54-484a-9d85-b44a9d3dfeab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6809","nonce":3097298224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6811","nonce":3097298224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6815","nonce":3097298224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6813","nonce":3097298224}]},"public_addr":"172.21.15.107:6809/3097298224","cluster_addr":"172.21.15.107:6811/3097298224","heartbeat_back_addr":"172.21.15.107:6815/3097298224","heartbeat_front_addr":"172.21.15.107:6813/3097298224","state":["exists","up"]},{"osd":3,"uuid":"54658731-2b9a-4d5e-ab13-7c1be7d34e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6820","nonce":3773076671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6824","nonce":3773076671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6831","nonce":3773076671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6828","nonce":3773076671}]},"public_addr":"172.21.15.107:6820/3773076671","cluster_addr":"172.21.15.107:6824/3773076671","heartbeat_back_addr":"172.21.15.107:6831/3773076671","heartbeat_front_addr":"172.21.15.107:6828/3773076671","state":["exists","up"]},{"osd":4,"uuid":"9a9dea13-8afe-485d-aa43-45eda691179b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6801","nonce":2747823542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6803","nonce":2747823542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6807","nonce":2747823542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6805","nonce":2747823542}]},"public_addr":"172.21.15.195:6801/2747823542","cluster_addr":"172.21.15.195:6803/2747823542","heartbeat_back_addr":"172.21.15.195:6807/2747823542","heartbeat_front_addr":"172.21.15.195:6805/2747823542","state":["exists","up"]},{"osd":5,"uuid":"db840d83-5a7d-468a-8aa2-1997945847c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6825","nonce":1484264540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6827","nonce":1484264540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6831","nonce":1484264540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6829","nonce":1484264540}]},"public_addr":"172.21.15.195:6825/1484264540","cluster_addr":"172.21.15.195:6827/1484264540","heartbeat_back_addr":"172.21.15.195:6831/1484264540","heartbeat_front_addr":"172.21.15.195:6829/1484264540","state":["exists","up"]},{"osd":6,"uuid":"38b5f165-bf91-467e-a615-4b4784337689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6811","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6813","nonce":2638471412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6817","nonce":2638471412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6823","nonce":2638471412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6821","nonce":2638471412}]},"public_addr":"172.21.15.195:6813/2638471412","cluster_addr":"172.21.15.195:6817/2638471412","heartbeat_back_addr":"172.21.15.195:6823/2638471412","heartbeat_front_addr":"172.21.15.195:6821/2638471412","state":["exists","up"]},{"osd":7,"uuid":"e5b3c596-494b-4df3-9437-bc2d52168ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6809","nonce":768189614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6812","nonce":768189614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6820","nonce":768189614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6816","nonce":768189614}]},"public_addr":"172.21.15.195:6809/768189614","cluster_addr":"172.21.15.195:6812/768189614","heartbeat_back_addr":"172.21.15.195:6820/768189614","heartbeat_front_addr":"172.21.15.195:6816/768189614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.209077+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.086601+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.221568+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.238524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.059477+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.148422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.079867+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.064916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T05:39:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.708+0000 7f763ee05640 1 -- 172.21.15.107:0/161208018 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7624066900 msgr2=0x7f7624068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.708+0000 7f763ee05640 1 --2- 172.21.15.107:0/161208018 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7624066900 0x7f7624068dc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f76381f0740 tx=0x7f7628008040 comp rx=0 tx=0).stop 2023-12-04T05:39:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.708+0000 7f763ee05640 1 -- 172.21.15.107:0/161208018 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638072110 msgr2=0x7f763813d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.708+0000 7f763ee05640 1 --2- 172.21.15.107:0/161208018 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638072110 0x7f763813d030 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f76340ad040 tx=0x7f76340192e0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.708+0000 7f763ee05640 1 -- 172.21.15.107:0/161208018 shutdown_connections 2023-12-04T05:39:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.708+0000 7f763ee05640 1 --2- 172.21.15.107:0/161208018 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7624066900 0x7f7624068dc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.708+0000 7f763ee05640 1 --2- 172.21.15.107:0/161208018 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f76380c56f0 0x7f76381626f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.708+0000 7f763ee05640 1 --2- 172.21.15.107:0/161208018 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f76380724f0 0x7f7638152930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.708+0000 7f763ee05640 1 --2- 172.21.15.107:0/161208018 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638072110 0x7f763813d030 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.708+0000 7f763ee05640 1 -- 172.21.15.107:0/161208018 >> 172.21.15.107:0/161208018 conn(0x7f76380c7ff0 msgr2=0x7f76380c92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:56.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.708+0000 7f763ee05640 1 -- 172.21.15.107:0/161208018 shutdown_connections 2023-12-04T05:39:56.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.708+0000 7f763ee05640 1 -- 172.21.15.107:0/161208018 wait complete. 2023-12-04T05:39:56.725 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-04T05:39:47.567702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-04T05:39:56.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-04T05:39:56.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f5425484640 1 Processor -- start 2023-12-04T05:39:56.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f5425484640 1 -- start start 2023-12-04T05:39:56.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f5425484640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54200c56f0 0x7f5420158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:56.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f5425484640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5420158da0 con 0x7f54200724f0 2023-12-04T05:39:56.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f5425484640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54201583d0 con 0x7f5420072110 2023-12-04T05:39:56.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f5425484640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54200c6170 con 0x7f54200c56f0 2023-12-04T05:39:56.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541e7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f54200724f0 0x7f5420140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53358/0 (socket says 172.21.15.107:53358) 2023-12-04T05:39:56.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541e7fc640 1 -- 172.21.15.107:0/2182694654 learned_addr learned my addr 172.21.15.107:0/2182694654 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:39:56.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541f7fe640 1 --2- 172.21.15.107:0/2182694654 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54200c56f0 0x7f5420158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:56.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541dffb640 1 -- 172.21.15.107:0/2182694654 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841781478 0 0) 0x7f5414001120 con 0x7f54200724f0 2023-12-04T05:39:56.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541dffb640 1 -- 172.21.15.107:0/2182694654 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5408003530 con 0x7f54200724f0 2023-12-04T05:39:56.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541dffb640 1 -- 172.21.15.107:0/2182694654 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3814343925 0 0) 0x7f5408003530 con 0x7f54200724f0 2023-12-04T05:39:56.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541dffb640 1 -- 172.21.15.107:0/2182694654 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54200c56f0 msgr2=0x7f5420158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:56.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541dffb640 1 --2- 172.21.15.107:0/2182694654 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54200c56f0 0x7f5420158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541dffb640 1 -- 172.21.15.107:0/2182694654 >> v1:172.21.15.195:6789/0 conn(0x7f5420072110 legacy=0x7f54200c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:56.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541dffb640 1 -- 172.21.15.107:0/2182694654 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5420157da0 con 0x7f54200724f0 2023-12-04T05:39:56.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541dffb640 1 -- 172.21.15.107:0/2182694654 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f5414001830 con 0x7f54200724f0 2023-12-04T05:39:56.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541dffb640 1 -- 172.21.15.107:0/2182694654 >> v1:172.21.15.107:6789/0 conn(0x7f54200724f0 legacy=0x7f5420140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:56.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541dffb640 1 --2- 172.21.15.107:0/2182694654 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54080039b0 0x7f5408003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:56.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541dffb640 1 --2- 172.21.15.107:0/2182694654 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54200c56f0 0x7f54080047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:56.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541dffb640 1 --2- 172.21.15.107:0/2182694654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5408004d10 0x7f5408005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:56.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541dffb640 1 -- 172.21.15.107:0/2182694654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54200c6170 con 0x7f5408004d10 2023-12-04T05:39:56.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541dffb640 1 -- 172.21.15.107:0/2182694654 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540800ba40 con 0x7f54080039b0 2023-12-04T05:39:56.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541dffb640 1 -- 172.21.15.107:0/2182694654 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540800bc00 con 0x7f54200c56f0 2023-12-04T05:39:56.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541f7fe640 1 --2- 172.21.15.107:0/2182694654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5408004d10 0x7f5408005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:56.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541effd640 1 --2- 172.21.15.107:0/2182694654 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54080039b0 0x7f5408003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:56.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541e7fc640 1 --2- 172.21.15.107:0/2182694654 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54200c56f0 0x7f54080047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:56.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541f7fe640 1 -- 172.21.15.107:0/2182694654 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54200c56f0 msgr2=0x7f54080047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:56.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541f7fe640 1 --2- 172.21.15.107:0/2182694654 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54200c56f0 0x7f54080047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541f7fe640 1 -- 172.21.15.107:0/2182694654 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54080039b0 msgr2=0x7f5408003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:56.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541f7fe640 1 --2- 172.21.15.107:0/2182694654 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54080039b0 0x7f5408003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541f7fe640 1 -- 172.21.15.107:0/2182694654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5408003530 con 0x7f5408004d10 2023-12-04T05:39:56.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.844+0000 7f541e7fc640 1 --2- 172.21.15.107:0/2182694654 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54200c56f0 0x7f54080047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:39:56.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.848+0000 7f541f7fe640 1 --2- 172.21.15.107:0/2182694654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5408004d10 0x7f5408005190 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5418009f90 tx=0x7f541801cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e3a82d36dddd55c server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:56.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.848+0000 7f541dffb640 1 -- 172.21.15.107:0/2182694654 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54180a5c30 con 0x7f5408004d10 2023-12-04T05:39:56.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.848+0000 7f541dffb640 1 -- 172.21.15.107:0/2182694654 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541800f040 con 0x7f5408004d10 2023-12-04T05:39:56.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.848+0000 7f541dffb640 1 -- 172.21.15.107:0/2182694654 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54180ad440 con 0x7f5408004d10 2023-12-04T05:39:56.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.848+0000 7f5425484640 1 -- 172.21.15.107:0/2182694654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5408004d10 msgr2=0x7f5408005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:56.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.848+0000 7f5425484640 1 --2- 172.21.15.107:0/2182694654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5408004d10 0x7f5408005190 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5418009f90 tx=0x7f541801cc20 comp rx=0 tx=0).stop 2023-12-04T05:39:56.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.848+0000 7f541f7fe640 1 -- 172.21.15.107:0/2182694654 reap_dead start 2023-12-04T05:39:56.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.848+0000 7f5425484640 1 -- 172.21.15.107:0/2182694654 shutdown_connections 2023-12-04T05:39:56.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.848+0000 7f5425484640 1 -- 172.21.15.107:0/2182694654 >> 172.21.15.107:0/2182694654 conn(0x7f54200c7ff0 msgr2=0x7f54200c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:56.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.848+0000 7f5425484640 1 -- 172.21.15.107:0/2182694654 shutdown_connections 2023-12-04T05:39:56.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.848+0000 7f5425484640 1 -- 172.21.15.107:0/2182694654 wait complete. 2023-12-04T05:39:56.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.848+0000 7f5425484640 1 Processor -- start 2023-12-04T05:39:56.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.848+0000 7f5425484640 1 -- start start 2023-12-04T05:39:56.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.848+0000 7f5425484640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5420155e30 0x7f54201ed410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:56.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.848+0000 7f5425484640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5420156210 0x7f54201ed950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:56.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.848+0000 7f5425484640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5420159340 0x7f542014cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:56.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.848+0000 7f5425484640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54201f06d0 con 0x7f5420155e30 2023-12-04T05:39:56.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.848+0000 7f5425484640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5420159850 con 0x7f5420156210 2023-12-04T05:39:56.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.848+0000 7f5425484640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54201f0950 con 0x7f5420159340 2023-12-04T05:39:56.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.852+0000 7f541effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5420155e30 0x7f54201ed410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:56.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.852+0000 7f541effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5420155e30 0x7f54201ed410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33890/0 (socket says 172.21.15.107:33890) 2023-12-04T05:39:56.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.852+0000 7f541effd640 1 -- 172.21.15.107:0/3817873585 learned_addr learned my addr 172.21.15.107:0/3817873585 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:39:56.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.852+0000 7f541f7fe640 1 --2- 172.21.15.107:0/3817873585 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5420159340 0x7f542014cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:56.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.852+0000 7f541effd640 1 -- 172.21.15.107:0/3817873585 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5420159340 msgr2=0x7f542014cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:56.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.852+0000 7f541effd640 1 --2- 172.21.15.107:0/3817873585 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5420159340 0x7f542014cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.852+0000 7f541e7fc640 1 --2- 172.21.15.107:0/3817873585 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5420156210 0x7f54201ed950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:56.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.852+0000 7f541effd640 1 -- 172.21.15.107:0/3817873585 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5420156210 msgr2=0x7f54201ed950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:56.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.852+0000 7f541effd640 1 --2- 172.21.15.107:0/3817873585 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5420156210 0x7f54201ed950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:56.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.852+0000 7f541effd640 1 -- 172.21.15.107:0/3817873585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54201f0c30 con 0x7f5420155e30 2023-12-04T05:39:56.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.852+0000 7f541effd640 1 --2- 172.21.15.107:0/3817873585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5420155e30 0x7f54201ed410 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f541000e550 tx=0x7f541000ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:56.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.852+0000 7f54077fe640 1 -- 172.21.15.107:0/3817873585 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54100052e0 con 0x7f5420155e30 2023-12-04T05:39:56.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.852+0000 7f54077fe640 1 -- 172.21.15.107:0/3817873585 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5410005440 con 0x7f5420155e30 2023-12-04T05:39:56.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.852+0000 7f5425484640 1 -- 172.21.15.107:0/3817873585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54201f0f10 con 0x7f5420155e30 2023-12-04T05:39:56.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.852+0000 7f54077fe640 1 -- 172.21.15.107:0/3817873585 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5410003820 con 0x7f5420155e30 2023-12-04T05:39:56.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.852+0000 7f5425484640 1 -- 172.21.15.107:0/3817873585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54201f14d0 con 0x7f5420155e30 2023-12-04T05:39:56.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.852+0000 7f5425484640 1 -- 172.21.15.107:0/3817873585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5420067870 con 0x7f5420155e30 2023-12-04T05:39:56.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.852+0000 7f54077fe640 1 -- 172.21.15.107:0/3817873585 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f5410003980 con 0x7f5420155e30 2023-12-04T05:39:56.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.852+0000 7f54077fe640 1 --2- 172.21.15.107:0/3817873585 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f540c066900 0x7f540c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:56.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.852+0000 7f541e7fc640 1 --2- 172.21.15.107:0/3817873585 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f540c066900 0x7f540c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:56.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.852+0000 7f54077fe640 1 -- 172.21.15.107:0/3817873585 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5410083f20 con 0x7f5420155e30 2023-12-04T05:39:56.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.852+0000 7f541e7fc640 1 --2- 172.21.15.107:0/3817873585 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f540c066900 0x7f540c068dc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5414000f70 tx=0x7f5414009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:56.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:56.856+0000 7f54077fe640 1 -- 172.21.15.107:0/3817873585 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f54100514b0 con 0x7f5420155e30 2023-12-04T05:39:57.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.000+0000 7f5425484640 1 -- 172.21.15.107:0/3817873585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f54200c2be0 con 0x7f5420155e30 2023-12-04T05:39:57.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.004+0000 7f54077fe640 1 -- 172.21.15.107:0/3817873585 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5410054ad0 con 0x7f5420155e30 2023-12-04T05:39:57.004 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-12-04T05:39:57.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.004+0000 7f5425484640 1 -- 172.21.15.107:0/3817873585 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f540c066900 msgr2=0x7f540c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:57.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.004+0000 7f5425484640 1 --2- 172.21.15.107:0/3817873585 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f540c066900 0x7f540c068dc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5414000f70 tx=0x7f5414009040 comp rx=0 tx=0).stop 2023-12-04T05:39:57.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.004+0000 7f5425484640 1 -- 172.21.15.107:0/3817873585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5420155e30 msgr2=0x7f54201ed410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:57.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.004+0000 7f5425484640 1 --2- 172.21.15.107:0/3817873585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5420155e30 0x7f54201ed410 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f541000e550 tx=0x7f541000ea10 comp rx=0 tx=0).stop 2023-12-04T05:39:57.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.004+0000 7f5425484640 1 -- 172.21.15.107:0/3817873585 shutdown_connections 2023-12-04T05:39:57.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.004+0000 7f5425484640 1 --2- 172.21.15.107:0/3817873585 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f540c066900 0x7f540c068dc0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.004+0000 7f5425484640 1 --2- 172.21.15.107:0/3817873585 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5420159340 0x7f542014cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.004+0000 7f5425484640 1 --2- 172.21.15.107:0/3817873585 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5420156210 0x7f54201ed950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.004+0000 7f5425484640 1 --2- 172.21.15.107:0/3817873585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5420155e30 0x7f54201ed410 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.004+0000 7f5425484640 1 -- 172.21.15.107:0/3817873585 >> 172.21.15.107:0/3817873585 conn(0x7f54200c7ff0 msgr2=0x7f5420157220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:57.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.004+0000 7f5425484640 1 -- 172.21.15.107:0/3817873585 shutdown_connections 2023-12-04T05:39:57.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.004+0000 7f5425484640 1 -- 172.21.15.107:0/3817873585 wait complete. 2023-12-04T05:39:57.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-04T05:39:57.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.124+0000 7f4aa74ed640 1 Processor -- start 2023-12-04T05:39:57.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.124+0000 7f4aa74ed640 1 -- start start 2023-12-04T05:39:57.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4aa74ed640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4aa0151180 0x7f4aa00c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4aa74ed640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa00700c0 con 0x7f4aa00714f0 2023-12-04T05:39:57.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4aa74ed640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa0156df0 con 0x7f4aa0070380 2023-12-04T05:39:57.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4aa74ed640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa01570d0 con 0x7f4aa0151180 2023-12-04T05:39:57.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4aa4a88640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4aa00714f0 0x7f4aa00718f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53362/0 (socket says 172.21.15.107:53362) 2023-12-04T05:39:57.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4aa4a88640 1 -- 172.21.15.107:0/2343248477 learned_addr learned my addr 172.21.15.107:0/2343248477 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:39:57.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4aa5289640 1 --2- 172.21.15.107:0/2343248477 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4aa0151180 0x7f4aa00c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4a9ffff640 1 -- 172.21.15.107:0/2343248477 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675761402 0 0) 0x7f4aa00700c0 con 0x7f4aa00714f0 2023-12-04T05:39:57.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4a9ffff640 1 -- 172.21.15.107:0/2343248477 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a88003530 con 0x7f4aa00714f0 2023-12-04T05:39:57.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4a9ffff640 1 -- 172.21.15.107:0/2343248477 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1733961378 0 0) 0x7f4a88003530 con 0x7f4aa00714f0 2023-12-04T05:39:57.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4a9ffff640 1 -- 172.21.15.107:0/2343248477 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4aa0151180 msgr2=0x7f4aa00c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:57.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4a9ffff640 1 --2- 172.21.15.107:0/2343248477 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4aa0151180 0x7f4aa00c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4a9ffff640 1 -- 172.21.15.107:0/2343248477 >> v1:172.21.15.195:6789/0 conn(0x7f4aa0070380 legacy=0x7f4aa0070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:57.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4a9ffff640 1 -- 172.21.15.107:0/2343248477 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa01573b0 con 0x7f4aa00714f0 2023-12-04T05:39:57.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4a9ffff640 1 -- 172.21.15.107:0/2343248477 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f4a94001a00 con 0x7f4aa00714f0 2023-12-04T05:39:57.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4a9ffff640 1 -- 172.21.15.107:0/2343248477 >> v1:172.21.15.107:6789/0 conn(0x7f4aa00714f0 legacy=0x7f4aa00718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:57.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4a9ffff640 1 --2- 172.21.15.107:0/2343248477 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4aa0151180 0x7f4a880039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4a9ffff640 1 --2- 172.21.15.107:0/2343248477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a88003f30 0x7f4a880047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4a9ffff640 1 --2- 172.21.15.107:0/2343248477 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4a88004d10 0x7f4a88005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4a9ffff640 1 -- 172.21.15.107:0/2343248477 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa01570d0 con 0x7f4a88003f30 2023-12-04T05:39:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4a9ffff640 1 -- 172.21.15.107:0/2343248477 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8800ba70 con 0x7f4a88004d10 2023-12-04T05:39:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4a9ffff640 1 -- 172.21.15.107:0/2343248477 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8800bc30 con 0x7f4aa0151180 2023-12-04T05:39:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4aa4a88640 1 --2- 172.21.15.107:0/2343248477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a88003f30 0x7f4a880047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4aa5a8a640 1 --2- 172.21.15.107:0/2343248477 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4a88004d10 0x7f4a88005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4aa5289640 1 --2- 172.21.15.107:0/2343248477 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4aa0151180 0x7f4a880039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4aa4a88640 1 -- 172.21.15.107:0/2343248477 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4aa0151180 msgr2=0x7f4a880039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4aa4a88640 1 --2- 172.21.15.107:0/2343248477 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4aa0151180 0x7f4a880039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4aa4a88640 1 -- 172.21.15.107:0/2343248477 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4a88004d10 msgr2=0x7f4a88005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4aa4a88640 1 --2- 172.21.15.107:0/2343248477 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4a88004d10 0x7f4a88005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.128+0000 7f4aa4a88640 1 -- 172.21.15.107:0/2343248477 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a88003530 con 0x7f4a88003f30 2023-12-04T05:39:57.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4aa4a88640 1 --2- 172.21.15.107:0/2343248477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a88003f30 0x7f4a880047d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4a9400a830 tx=0x7f4a9401faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2842e8206020677e server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:57.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4a9ffff640 1 -- 172.21.15.107:0/2343248477 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a940aa6d0 con 0x7f4a88003f30 2023-12-04T05:39:57.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4a9ffff640 1 -- 172.21.15.107:0/2343248477 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a940aa830 con 0x7f4a88003f30 2023-12-04T05:39:57.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4a9ffff640 1 -- 172.21.15.107:0/2343248477 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a940aac60 con 0x7f4a88003f30 2023-12-04T05:39:57.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4aa74ed640 1 -- 172.21.15.107:0/2343248477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a88003f30 msgr2=0x7f4a880047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:57.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4aa74ed640 1 --2- 172.21.15.107:0/2343248477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a88003f30 0x7f4a880047d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4a9400a830 tx=0x7f4a9401faa0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4aa5a8a640 1 -- 172.21.15.107:0/2343248477 reap_dead start 2023-12-04T05:39:57.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4aa74ed640 1 -- 172.21.15.107:0/2343248477 shutdown_connections 2023-12-04T05:39:57.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4aa74ed640 1 -- 172.21.15.107:0/2343248477 >> 172.21.15.107:0/2343248477 conn(0x7f4aa00c7ff0 msgr2=0x7f4aa00c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:57.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4aa74ed640 1 -- 172.21.15.107:0/2343248477 shutdown_connections 2023-12-04T05:39:57.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4aa74ed640 1 -- 172.21.15.107:0/2343248477 wait complete. 2023-12-04T05:39:57.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4aa74ed640 1 Processor -- start 2023-12-04T05:39:57.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4aa74ed640 1 -- start start 2023-12-04T05:39:57.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4aa74ed640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4aa0141a70 0x7f4aa00c4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4aa74ed640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4aa015ab40 0x7f4aa0141e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4aa74ed640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4aa0157e30 0x7f4aa01ee350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4aa74ed640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa01582b0 con 0x7f4aa0157e30 2023-12-04T05:39:57.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4aa74ed640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa01edec0 con 0x7f4aa0141a70 2023-12-04T05:39:57.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4aa74ed640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa01f1a90 con 0x7f4aa015ab40 2023-12-04T05:39:57.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4aa5a8a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4aa0157e30 0x7f4aa01ee350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4aa5a8a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4aa0157e30 0x7f4aa01ee350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33908/0 (socket says 172.21.15.107:33908) 2023-12-04T05:39:57.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4aa5a8a640 1 -- 172.21.15.107:0/289797256 learned_addr learned my addr 172.21.15.107:0/289797256 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:39:57.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.132+0000 7f4aa4a88640 1 --2- 172.21.15.107:0/289797256 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4aa015ab40 0x7f4aa0141e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.136+0000 7f4aa5289640 1 --2- 172.21.15.107:0/289797256 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4aa0141a70 0x7f4aa00c4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.136+0000 7f4aa5a8a640 1 -- 172.21.15.107:0/289797256 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4aa015ab40 msgr2=0x7f4aa0141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:57.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.136+0000 7f4aa5a8a640 1 --2- 172.21.15.107:0/289797256 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4aa015ab40 0x7f4aa0141e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.136+0000 7f4aa5a8a640 1 -- 172.21.15.107:0/289797256 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4aa0141a70 msgr2=0x7f4aa00c4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:57.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.136+0000 7f4aa5a8a640 1 --2- 172.21.15.107:0/289797256 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4aa0141a70 0x7f4aa00c4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.136+0000 7f4aa5a8a640 1 -- 172.21.15.107:0/289797256 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa006e570 con 0x7f4aa0157e30 2023-12-04T05:39:57.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.136+0000 7f4aa5289640 1 --2- 172.21.15.107:0/289797256 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4aa0141a70 0x7f4aa00c4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:39:57.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.136+0000 7f4aa5a8a640 1 --2- 172.21.15.107:0/289797256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4aa0157e30 0x7f4aa01ee350 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4a90000c10 tx=0x7f4a9000e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:57.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.136+0000 7f4a9dffb640 1 -- 172.21.15.107:0/289797256 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a900052e0 con 0x7f4aa0157e30 2023-12-04T05:39:57.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.136+0000 7f4a9dffb640 1 -- 172.21.15.107:0/289797256 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a90005460 con 0x7f4aa0157e30 2023-12-04T05:39:57.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.136+0000 7f4aa74ed640 1 -- 172.21.15.107:0/289797256 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa00c4670 con 0x7f4aa0157e30 2023-12-04T05:39:57.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.136+0000 7f4a9dffb640 1 -- 172.21.15.107:0/289797256 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a90003840 con 0x7f4aa0157e30 2023-12-04T05:39:57.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.136+0000 7f4aa74ed640 1 -- 172.21.15.107:0/289797256 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa01edb50 con 0x7f4aa0157e30 2023-12-04T05:39:57.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.136+0000 7f4aa74ed640 1 -- 172.21.15.107:0/289797256 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa01ee030 con 0x7f4aa0157e30 2023-12-04T05:39:57.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.136+0000 7f4a9dffb640 1 -- 172.21.15.107:0/289797256 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4a900039a0 con 0x7f4aa0157e30 2023-12-04T05:39:57.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.136+0000 7f4a9dffb640 1 --2- 172.21.15.107:0/289797256 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4a8c066900 0x7f4a8c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.136+0000 7f4a9dffb640 1 -- 172.21.15.107:0/289797256 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4a90083830 con 0x7f4aa0157e30 2023-12-04T05:39:57.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.136+0000 7f4aa5289640 1 --2- 172.21.15.107:0/289797256 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4a8c066900 0x7f4a8c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.136+0000 7f4aa5289640 1 --2- 172.21.15.107:0/289797256 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4a8c066900 0x7f4a8c068dc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4a98009a00 tx=0x7f4a98008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:57.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.140+0000 7f4a9dffb640 1 -- 172.21.15.107:0/289797256 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4a90050dc0 con 0x7f4aa0157e30 2023-12-04T05:39:57.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.295+0000 7f4aa74ed640 1 -- 172.21.15.107:0/289797256 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4aa0067870 con 0x7f4aa0157e30 2023-12-04T05:39:57.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.295+0000 7f4a9dffb640 1 -- 172.21.15.107:0/289797256 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7f4a90019070 con 0x7f4aa0157e30 2023-12-04T05:39:57.298 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:39:57.298 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-12-04T05:39:57.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-04T05:39:57.299 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2023-12-04T05:39:57.299 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2023-12-04T05:39:57.299 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-04T05:39:57.299 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-12-04T05:39:57.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.299+0000 7f4aa74ed640 1 -- 172.21.15.107:0/289797256 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4a8c066900 msgr2=0x7f4a8c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:57.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.299+0000 7f4aa74ed640 1 --2- 172.21.15.107:0/289797256 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4a8c066900 0x7f4a8c068dc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4a98009a00 tx=0x7f4a98008040 comp rx=0 tx=0).stop 2023-12-04T05:39:57.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.299+0000 7f4aa74ed640 1 -- 172.21.15.107:0/289797256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4aa0157e30 msgr2=0x7f4aa01ee350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:57.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.299+0000 7f4aa74ed640 1 --2- 172.21.15.107:0/289797256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4aa0157e30 0x7f4aa01ee350 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4a90000c10 tx=0x7f4a9000e8a0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.299+0000 7f4aa74ed640 1 -- 172.21.15.107:0/289797256 shutdown_connections 2023-12-04T05:39:57.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.299+0000 7f4aa74ed640 1 --2- 172.21.15.107:0/289797256 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4a8c066900 0x7f4a8c068dc0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.299+0000 7f4aa74ed640 1 --2- 172.21.15.107:0/289797256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4aa0157e30 0x7f4aa01ee350 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.299+0000 7f4aa74ed640 1 --2- 172.21.15.107:0/289797256 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4aa015ab40 0x7f4aa0141e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.299+0000 7f4aa74ed640 1 --2- 172.21.15.107:0/289797256 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4aa0141a70 0x7f4aa00c4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.299+0000 7f4aa74ed640 1 -- 172.21.15.107:0/289797256 >> 172.21.15.107:0/289797256 conn(0x7f4aa00c7ff0 msgr2=0x7f4aa00718d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:57.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.299+0000 7f4aa74ed640 1 -- 172.21.15.107:0/289797256 shutdown_connections 2023-12-04T05:39:57.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.299+0000 7f4aa74ed640 1 -- 172.21.15.107:0/289797256 wait complete. 2023-12-04T05:39:57.316 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-04T05:39:57.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-04T05:39:57.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54f724c640 1 Processor -- start 2023-12-04T05:39:57.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54f724c640 1 -- start start 2023-12-04T05:39:57.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54f724c640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54f0153ff0 0x7f54f00ca420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54f724c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f0159cf0 con 0x7f54f00ca960 2023-12-04T05:39:57.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54f724c640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f01543d0 con 0x7f54f00cbb00 2023-12-04T05:39:57.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54f724c640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54f0159f70 con 0x7f54f0153ff0 2023-12-04T05:39:57.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54effff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f54f00ca960 0x7f54f00cafc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53366/0 (socket says 172.21.15.107:53366) 2023-12-04T05:39:57.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54effff640 1 -- 172.21.15.107:0/3537559715 learned_addr learned my addr 172.21.15.107:0/3537559715 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:39:57.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54f4fe8640 1 --2- 172.21.15.107:0/3537559715 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54f0153ff0 0x7f54f00ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54ef7fe640 1 -- 172.21.15.107:0/3537559715 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899479746 0 0) 0x7f54f0159cf0 con 0x7f54f00ca960 2023-12-04T05:39:57.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54ef7fe640 1 -- 172.21.15.107:0/3537559715 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54d8003530 con 0x7f54f00ca960 2023-12-04T05:39:57.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54ef7fe640 1 -- 172.21.15.107:0/3537559715 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4182126885 0 0) 0x7f54d8003530 con 0x7f54f00ca960 2023-12-04T05:39:57.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54ef7fe640 1 -- 172.21.15.107:0/3537559715 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54f0153ff0 msgr2=0x7f54f00ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:57.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54ef7fe640 1 --2- 172.21.15.107:0/3537559715 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54f0153ff0 0x7f54f00ca420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54ef7fe640 1 -- 172.21.15.107:0/3537559715 >> v1:172.21.15.195:6789/0 conn(0x7f54f00cbb00 legacy=0x7f54f015d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:57.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54ef7fe640 1 -- 172.21.15.107:0/3537559715 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f015a250 con 0x7f54f00ca960 2023-12-04T05:39:57.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54ef7fe640 1 -- 172.21.15.107:0/3537559715 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f54e4001680 con 0x7f54f00ca960 2023-12-04T05:39:57.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54ef7fe640 1 -- 172.21.15.107:0/3537559715 >> v1:172.21.15.107:6789/0 conn(0x7f54f00ca960 legacy=0x7f54f00cafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:57.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54ef7fe640 1 --2- 172.21.15.107:0/3537559715 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54f0153ff0 0x7f54d80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54ef7fe640 1 --2- 172.21.15.107:0/3537559715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54d8003f30 0x7f54d80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54ef7fe640 1 --2- 172.21.15.107:0/3537559715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54d8004d10 0x7f54d8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54ef7fe640 1 -- 172.21.15.107:0/3537559715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f0159f70 con 0x7f54d8004d10 2023-12-04T05:39:57.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54ef7fe640 1 -- 172.21.15.107:0/3537559715 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d800bb00 con 0x7f54d8003f30 2023-12-04T05:39:57.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54ef7fe640 1 -- 172.21.15.107:0/3537559715 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d800bcc0 con 0x7f54f0153ff0 2023-12-04T05:39:57.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54f57e9640 1 --2- 172.21.15.107:0/3537559715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54d8004d10 0x7f54d8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54effff640 1 --2- 172.21.15.107:0/3537559715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54d8003f30 0x7f54d80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54f4fe8640 1 --2- 172.21.15.107:0/3537559715 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54f0153ff0 0x7f54d80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54f57e9640 1 -- 172.21.15.107:0/3537559715 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54f0153ff0 msgr2=0x7f54d80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:57.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54f57e9640 1 --2- 172.21.15.107:0/3537559715 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54f0153ff0 0x7f54d80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54f57e9640 1 -- 172.21.15.107:0/3537559715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54d8003f30 msgr2=0x7f54d80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:57.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54f57e9640 1 --2- 172.21.15.107:0/3537559715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54d8003f30 0x7f54d80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.435+0000 7f54f57e9640 1 -- 172.21.15.107:0/3537559715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d8003530 con 0x7f54d8004d10 2023-12-04T05:39:57.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.439+0000 7f54f57e9640 1 --2- 172.21.15.107:0/3537559715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54d8004d10 0x7f54d8005190 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f54e8001e40 tx=0x7f54e801c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd1abef5d99a0cc server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:57.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.439+0000 7f54ef7fe640 1 -- 172.21.15.107:0/3537559715 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e8003b80 con 0x7f54d8004d10 2023-12-04T05:39:57.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.439+0000 7f54ef7fe640 1 -- 172.21.15.107:0/3537559715 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54e8003ce0 con 0x7f54d8004d10 2023-12-04T05:39:57.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.439+0000 7f54ef7fe640 1 -- 172.21.15.107:0/3537559715 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e80052c0 con 0x7f54d8004d10 2023-12-04T05:39:57.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.439+0000 7f54f724c640 1 -- 172.21.15.107:0/3537559715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54d8004d10 msgr2=0x7f54d8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:57.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.439+0000 7f54f724c640 1 --2- 172.21.15.107:0/3537559715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54d8004d10 0x7f54d8005190 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f54e8001e40 tx=0x7f54e801c740 comp rx=0 tx=0).stop 2023-12-04T05:39:57.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.439+0000 7f54f57e9640 1 -- 172.21.15.107:0/3537559715 reap_dead start 2023-12-04T05:39:57.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.439+0000 7f54f724c640 1 -- 172.21.15.107:0/3537559715 shutdown_connections 2023-12-04T05:39:57.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.439+0000 7f54f724c640 1 -- 172.21.15.107:0/3537559715 >> 172.21.15.107:0/3537559715 conn(0x7f54f006c100 msgr2=0x7f54f006c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:57.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.439+0000 7f54f724c640 1 -- 172.21.15.107:0/3537559715 shutdown_connections 2023-12-04T05:39:57.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.439+0000 7f54f724c640 1 -- 172.21.15.107:0/3537559715 wait complete. 2023-12-04T05:39:57.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54f724c640 1 Processor -- start 2023-12-04T05:39:57.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54f724c640 1 -- start start 2023-12-04T05:39:57.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54f724c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54d8004d10 0x7f54f015c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54f724c640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54f01523b0 0x7f54f0152810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54f724c640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54f01f30b0 0x7f54f01f3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54f724c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f01532c0 con 0x7f54d8004d10 2023-12-04T05:39:57.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54f724c640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f01f3a50 con 0x7f54f01523b0 2023-12-04T05:39:57.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54f724c640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54f01534f0 con 0x7f54f01f30b0 2023-12-04T05:39:57.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54f4fe8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54d8004d10 0x7f54f015c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54f4fe8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54d8004d10 0x7f54f015c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33928/0 (socket says 172.21.15.107:33928) 2023-12-04T05:39:57.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54f4fe8640 1 -- 172.21.15.107:0/3327788178 learned_addr learned my addr 172.21.15.107:0/3327788178 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:39:57.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54effff640 1 --2- 172.21.15.107:0/3327788178 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54f01523b0 0x7f54f0152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54f57e9640 1 --2- 172.21.15.107:0/3327788178 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54f01f30b0 0x7f54f01f3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54f4fe8640 1 -- 172.21.15.107:0/3327788178 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54f01f30b0 msgr2=0x7f54f01f3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:57.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54f4fe8640 1 --2- 172.21.15.107:0/3327788178 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54f01f30b0 0x7f54f01f3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54f4fe8640 1 -- 172.21.15.107:0/3327788178 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54f01523b0 msgr2=0x7f54f0152810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:57.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54f4fe8640 1 --2- 172.21.15.107:0/3327788178 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54f01523b0 0x7f54f0152810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54f4fe8640 1 -- 172.21.15.107:0/3327788178 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f01e3f30 con 0x7f54d8004d10 2023-12-04T05:39:57.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54effff640 1 --2- 172.21.15.107:0/3327788178 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54f01523b0 0x7f54f0152810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:39:57.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54f4fe8640 1 --2- 172.21.15.107:0/3327788178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54d8004d10 0x7f54f015c4a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f54e00115e0 tx=0x7f54e0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:57.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54ed7fa640 1 -- 172.21.15.107:0/3327788178 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e000f8f0 con 0x7f54d8004d10 2023-12-04T05:39:57.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54ed7fa640 1 -- 172.21.15.107:0/3327788178 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54e000fa50 con 0x7f54d8004d10 2023-12-04T05:39:57.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54ed7fa640 1 -- 172.21.15.107:0/3327788178 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e000fdc0 con 0x7f54d8004d10 2023-12-04T05:39:57.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54f724c640 1 -- 172.21.15.107:0/3327788178 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f01e4210 con 0x7f54d8004d10 2023-12-04T05:39:57.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54f724c640 1 -- 172.21.15.107:0/3327788178 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f015ce90 con 0x7f54d8004d10 2023-12-04T05:39:57.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.443+0000 7f54f724c640 1 -- 172.21.15.107:0/3327788178 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f0067870 con 0x7f54d8004d10 2023-12-04T05:39:57.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.447+0000 7f54ed7fa640 1 -- 172.21.15.107:0/3327788178 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f54e0015460 con 0x7f54d8004d10 2023-12-04T05:39:57.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.447+0000 7f54ed7fa640 1 --2- 172.21.15.107:0/3327788178 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f54dc0668d0 0x7f54dc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.447+0000 7f54ed7fa640 1 -- 172.21.15.107:0/3327788178 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f54e0084530 con 0x7f54d8004d10 2023-12-04T05:39:57.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.447+0000 7f54effff640 1 --2- 172.21.15.107:0/3327788178 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f54dc0668d0 0x7f54dc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.447+0000 7f54effff640 1 --2- 172.21.15.107:0/3327788178 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f54dc0668d0 0x7f54dc068d90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f54e4001ec0 tx=0x7f54e4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:57.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.451+0000 7f54ed7fa640 1 -- 172.21.15.107:0/3327788178 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f54e0051b50 con 0x7f54d8004d10 2023-12-04T05:39:57.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.603+0000 7f54f724c640 1 -- 172.21.15.107:0/3327788178 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f54f015d2b0 con 0x7f54d8004d10 2023-12-04T05:39:57.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.683+0000 7f54ed7fa640 1 -- 172.21.15.107:0/3327788178 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7f54e0055170 con 0x7f54d8004d10 2023-12-04T05:39:57.685 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' created 2023-12-04T05:39:57.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.687+0000 7f54f724c640 1 -- 172.21.15.107:0/3327788178 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f54dc0668d0 msgr2=0x7f54dc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:57.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.687+0000 7f54f724c640 1 --2- 172.21.15.107:0/3327788178 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f54dc0668d0 0x7f54dc068d90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f54e4001ec0 tx=0x7f54e4009040 comp rx=0 tx=0).stop 2023-12-04T05:39:57.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.687+0000 7f54f724c640 1 -- 172.21.15.107:0/3327788178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54d8004d10 msgr2=0x7f54f015c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:57.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.687+0000 7f54f724c640 1 --2- 172.21.15.107:0/3327788178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54d8004d10 0x7f54f015c4a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f54e00115e0 tx=0x7f54e0011aa0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.687+0000 7f54f724c640 1 -- 172.21.15.107:0/3327788178 shutdown_connections 2023-12-04T05:39:57.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.687+0000 7f54f724c640 1 --2- 172.21.15.107:0/3327788178 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f54dc0668d0 0x7f54dc068d90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.687+0000 7f54f724c640 1 --2- 172.21.15.107:0/3327788178 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54f01f30b0 0x7f54f01f3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.687+0000 7f54f724c640 1 --2- 172.21.15.107:0/3327788178 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54f01523b0 0x7f54f0152810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.687+0000 7f54f724c640 1 --2- 172.21.15.107:0/3327788178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54d8004d10 0x7f54f015c4a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.687+0000 7f54f724c640 1 -- 172.21.15.107:0/3327788178 >> 172.21.15.107:0/3327788178 conn(0x7f54f006c100 msgr2=0x7f54f006c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:57.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.687+0000 7f54f724c640 1 -- 172.21.15.107:0/3327788178 shutdown_connections 2023-12-04T05:39:57.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.687+0000 7f54f724c640 1 -- 172.21.15.107:0/3327788178 wait complete. 2023-12-04T05:39:57.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-04T05:39:57.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f5325c7a640 1 Processor -- start 2023-12-04T05:39:57.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f5325c7a640 1 -- start start 2023-12-04T05:39:57.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f5325c7a640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5320072110 0x7f5320158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f5325c7a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5320158da0 con 0x7f53200c56f0 2023-12-04T05:39:57.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f5325c7a640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53201583d0 con 0x7f53200724f0 2023-12-04T05:39:57.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f5325c7a640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53200c6170 con 0x7f5320072110 2023-12-04T05:39:57.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f53200c56f0 0x7f5320140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60250/0 (socket says 172.21.15.107:60250) 2023-12-04T05:39:57.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531ffff640 1 -- 172.21.15.107:0/2707420395 learned_addr learned my addr 172.21.15.107:0/2707420395 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:39:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531f7fe640 1 --2- 172.21.15.107:0/2707420395 >> v2:172.21.15.195:3300/0 conn(0x7f5320072110 0x7f5320158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531e7fc640 1 -- 172.21.15.107:0/2707420395 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252241618 0 0) 0x7f5310001120 con 0x7f53200c56f0 2023-12-04T05:39:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531e7fc640 1 -- 172.21.15.107:0/2707420395 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f530c003530 con 0x7f53200c56f0 2023-12-04T05:39:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531e7fc640 1 -- 172.21.15.107:0/2707420395 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1290428680 0 0) 0x7f530c003530 con 0x7f53200c56f0 2023-12-04T05:39:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531e7fc640 1 -- 172.21.15.107:0/2707420395 >> v2:172.21.15.195:3300/0 conn(0x7f5320072110 msgr2=0x7f5320158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531e7fc640 1 --2- 172.21.15.107:0/2707420395 >> v2:172.21.15.195:3300/0 conn(0x7f5320072110 0x7f5320158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531e7fc640 1 -- 172.21.15.107:0/2707420395 >> v1:172.21.15.195:6789/0 conn(0x7f53200724f0 legacy=0x7f53200c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531e7fc640 1 -- 172.21.15.107:0/2707420395 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5320157da0 con 0x7f53200c56f0 2023-12-04T05:39:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531e7fc640 1 -- 172.21.15.107:0/2707420395 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f5310001810 con 0x7f53200c56f0 2023-12-04T05:39:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531e7fc640 1 -- 172.21.15.107:0/2707420395 >> v1:172.21.15.107:6789/0 conn(0x7f53200c56f0 legacy=0x7f5320140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531e7fc640 1 --2- 172.21.15.107:0/2707420395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f530c003980 0x7f530c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531e7fc640 1 --2- 172.21.15.107:0/2707420395 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f530c0047d0 0x7f530c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531e7fc640 1 --2- 172.21.15.107:0/2707420395 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f530c005190 0x7f530c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531e7fc640 1 -- 172.21.15.107:0/2707420395 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53200c6170 con 0x7f530c003980 2023-12-04T05:39:57.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531e7fc640 1 -- 172.21.15.107:0/2707420395 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530c00c040 con 0x7f530c0047d0 2023-12-04T05:39:57.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531e7fc640 1 -- 172.21.15.107:0/2707420395 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f530c00c200 con 0x7f530c005190 2023-12-04T05:39:57.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531f7fe640 1 --2- 172.21.15.107:0/2707420395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f530c003980 0x7f530c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531f7fe640 1 -- 172.21.15.107:0/2707420395 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f530c005190 msgr2=0x7f530c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:57.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531f7fe640 1 --2- 172.21.15.107:0/2707420395 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f530c005190 0x7f530c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531f7fe640 1 -- 172.21.15.107:0/2707420395 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f530c0047d0 msgr2=0x7f530c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:57.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531effd640 1 --2- 172.21.15.107:0/2707420395 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f530c0047d0 0x7f530c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531f7fe640 1 --2- 172.21.15.107:0/2707420395 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f530c0047d0 0x7f530c004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531ffff640 1 -- 172.21.15.107:0/2707420395 reap_dead start 2023-12-04T05:39:57.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.835+0000 7f531f7fe640 1 -- 172.21.15.107:0/2707420395 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530c003530 con 0x7f530c003980 2023-12-04T05:39:57.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.839+0000 7f531f7fe640 1 --2- 172.21.15.107:0/2707420395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f530c003980 0x7f530c003da0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5318009ae0 tx=0x7f531801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d31800db4d63faf8 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:57.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.839+0000 7f531e7fc640 1 -- 172.21.15.107:0/2707420395 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53180a5640 con 0x7f530c003980 2023-12-04T05:39:57.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.839+0000 7f531e7fc640 1 -- 172.21.15.107:0/2707420395 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53180a57a0 con 0x7f530c003980 2023-12-04T05:39:57.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.839+0000 7f531e7fc640 1 -- 172.21.15.107:0/2707420395 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53180a5b50 con 0x7f530c003980 2023-12-04T05:39:57.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.839+0000 7f5325c7a640 1 -- 172.21.15.107:0/2707420395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f530c003980 msgr2=0x7f530c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:57.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.839+0000 7f5325c7a640 1 --2- 172.21.15.107:0/2707420395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f530c003980 0x7f530c003da0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5318009ae0 tx=0x7f531801ca50 comp rx=0 tx=0).stop 2023-12-04T05:39:57.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.839+0000 7f5325c7a640 1 -- 172.21.15.107:0/2707420395 shutdown_connections 2023-12-04T05:39:57.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.839+0000 7f5325c7a640 1 --2- 172.21.15.107:0/2707420395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f530c003980 0x7f530c003da0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.839+0000 7f5325c7a640 1 -- 172.21.15.107:0/2707420395 >> 172.21.15.107:0/2707420395 conn(0x7f53200c7ff0 msgr2=0x7f53200c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:57.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.839+0000 7f5325c7a640 1 -- 172.21.15.107:0/2707420395 shutdown_connections 2023-12-04T05:39:57.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.839+0000 7f5325c7a640 1 -- 172.21.15.107:0/2707420395 wait complete. 2023-12-04T05:39:57.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.839+0000 7f5325c7a640 1 Processor -- start 2023-12-04T05:39:57.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.839+0000 7f5325c7a640 1 -- start start 2023-12-04T05:39:57.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f5325c7a640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f530c003980 0x7f532015ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f5325c7a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f532015a8e0 0x7f53201eddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f5325c7a640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f53201599b0 0x7f53200c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f5325c7a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5320150d30 con 0x7f532015a8e0 2023-12-04T05:39:57.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f5325c7a640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5320158f60 con 0x7f530c003980 2023-12-04T05:39:57.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f5325c7a640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532015b900 con 0x7f53201599b0 2023-12-04T05:39:57.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f531effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f532015a8e0 0x7f53201eddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f531effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f532015a8e0 0x7f53201eddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33690/0 (socket says 172.21.15.107:33690) 2023-12-04T05:39:57.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f531effd640 1 -- 172.21.15.107:0/1844597925 learned_addr learned my addr 172.21.15.107:0/1844597925 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:39:57.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f531ffff640 1 --2- 172.21.15.107:0/1844597925 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f53201599b0 0x7f53200c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f531f7fe640 1 --2- 172.21.15.107:0/1844597925 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f530c003980 0x7f532015ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f531effd640 1 -- 172.21.15.107:0/1844597925 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f53201599b0 msgr2=0x7f53200c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:57.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f531effd640 1 --2- 172.21.15.107:0/1844597925 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f53201599b0 0x7f53200c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f531effd640 1 -- 172.21.15.107:0/1844597925 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f530c003980 msgr2=0x7f532015ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:57.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f531effd640 1 --2- 172.21.15.107:0/1844597925 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f530c003980 0x7f532015ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:57.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f531effd640 1 -- 172.21.15.107:0/1844597925 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f532015bbe0 con 0x7f532015a8e0 2023-12-04T05:39:57.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f531effd640 1 --2- 172.21.15.107:0/1844597925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f532015a8e0 0x7f53201eddf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f530800d8e0 tx=0x7f530800dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:57.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f5317fff640 1 -- 172.21.15.107:0/1844597925 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5308003db0 con 0x7f532015a8e0 2023-12-04T05:39:57.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f5317fff640 1 -- 172.21.15.107:0/1844597925 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5308005070 con 0x7f532015a8e0 2023-12-04T05:39:57.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f5317fff640 1 -- 172.21.15.107:0/1844597925 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5308005420 con 0x7f532015a8e0 2023-12-04T05:39:57.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f5325c7a640 1 -- 172.21.15.107:0/1844597925 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f532015bec0 con 0x7f532015a8e0 2023-12-04T05:39:57.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f5325c7a640 1 -- 172.21.15.107:0/1844597925 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53200ca400 con 0x7f532015a8e0 2023-12-04T05:39:57.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f5325c7a640 1 -- 172.21.15.107:0/1844597925 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53201596d0 con 0x7f532015a8e0 2023-12-04T05:39:57.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f5317fff640 1 -- 172.21.15.107:0/1844597925 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f5308005580 con 0x7f532015a8e0 2023-12-04T05:39:57.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f5317fff640 1 --2- 172.21.15.107:0/1844597925 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f53000668d0 0x7f5300068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:57.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f5317fff640 1 -- 172.21.15.107:0/1844597925 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f5308084e10 con 0x7f532015a8e0 2023-12-04T05:39:57.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f531f7fe640 1 --2- 172.21.15.107:0/1844597925 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f53000668d0 0x7f5300068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:57.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.843+0000 7f531f7fe640 1 --2- 172.21.15.107:0/1844597925 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f53000668d0 0x7f5300068d90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f53180af000 tx=0x7f5318008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:57.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.847+0000 7f5317fff640 1 -- 172.21.15.107:0/1844597925 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5308052310 con 0x7f532015a8e0 2023-12-04T05:39:57.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:57.991+0000 7f5325c7a640 1 -- 172.21.15.107:0/1844597925 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f53201596d0 con 0x7f532015a8e0 2023-12-04T05:39:58.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.687+0000 7f5317fff640 1 -- 172.21.15.107:0/1844597925 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) v1 ==== 170+0+0 (secure 0 0 0) 0x7f53201596d0 con 0x7f532015a8e0 2023-12-04T05:39:58.690 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' created 2023-12-04T05:39:58.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.691+0000 7f5325c7a640 1 -- 172.21.15.107:0/1844597925 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f53000668d0 msgr2=0x7f5300068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:58.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.691+0000 7f5325c7a640 1 --2- 172.21.15.107:0/1844597925 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f53000668d0 0x7f5300068d90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f53180af000 tx=0x7f5318008040 comp rx=0 tx=0).stop 2023-12-04T05:39:58.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.691+0000 7f5325c7a640 1 -- 172.21.15.107:0/1844597925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f532015a8e0 msgr2=0x7f53201eddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:58.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.691+0000 7f5325c7a640 1 --2- 172.21.15.107:0/1844597925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f532015a8e0 0x7f53201eddf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f530800d8e0 tx=0x7f530800dda0 comp rx=0 tx=0).stop 2023-12-04T05:39:58.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.691+0000 7f5325c7a640 1 -- 172.21.15.107:0/1844597925 shutdown_connections 2023-12-04T05:39:58.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.691+0000 7f5325c7a640 1 --2- 172.21.15.107:0/1844597925 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f53000668d0 0x7f5300068d90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:58.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.691+0000 7f5325c7a640 1 --2- 172.21.15.107:0/1844597925 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f53201599b0 0x7f53200c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:58.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.691+0000 7f5325c7a640 1 --2- 172.21.15.107:0/1844597925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f532015a8e0 0x7f53201eddf0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:58.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.691+0000 7f5325c7a640 1 --2- 172.21.15.107:0/1844597925 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f530c003980 0x7f532015ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:58.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.691+0000 7f5325c7a640 1 -- 172.21.15.107:0/1844597925 >> 172.21.15.107:0/1844597925 conn(0x7f53200c7ff0 msgr2=0x7f53200c6310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:58.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.691+0000 7f5325c7a640 1 -- 172.21.15.107:0/1844597925 shutdown_connections 2023-12-04T05:39:58.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.691+0000 7f5325c7a640 1 -- 172.21.15.107:0/1844597925 wait complete. 2023-12-04T05:39:58.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-04T05:39:58.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46fca94640 1 Processor -- start 2023-12-04T05:39:58.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46fca94640 1 -- start start 2023-12-04T05:39:58.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46fca94640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f46f80b1b80 0x7f46f80c2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:58.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46fca94640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f80c7a40 con 0x7f46f8157e10 2023-12-04T05:39:58.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46fca94640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f806c120 con 0x7f46f80c5c60 2023-12-04T05:39:58.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46fca94640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46f80c6b50 con 0x7f46f80b1b80 2023-12-04T05:39:58.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f6d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f46f8157e10 0x7f46f81626b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60252/0 (socket says 172.21.15.107:60252) 2023-12-04T05:39:58.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f6d9d640 1 -- 172.21.15.107:0/3563079352 learned_addr learned my addr 172.21.15.107:0/3563079352 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:39:58.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f659c640 1 --2- 172.21.15.107:0/3563079352 >> v2:172.21.15.195:3300/0 conn(0x7f46f80b1b80 0x7f46f80c2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:58.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f559a640 1 -- 172.21.15.107:0/3563079352 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858524549 0 0) 0x7f46f80c7a40 con 0x7f46f8157e10 2023-12-04T05:39:58.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f559a640 1 -- 172.21.15.107:0/3563079352 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46e4003530 con 0x7f46f8157e10 2023-12-04T05:39:58.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f559a640 1 -- 172.21.15.107:0/3563079352 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1883651631 0 0) 0x7f46e4003530 con 0x7f46f8157e10 2023-12-04T05:39:58.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f559a640 1 -- 172.21.15.107:0/3563079352 >> v2:172.21.15.195:3300/0 conn(0x7f46f80b1b80 msgr2=0x7f46f80c2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:58.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f559a640 1 --2- 172.21.15.107:0/3563079352 >> v2:172.21.15.195:3300/0 conn(0x7f46f80b1b80 0x7f46f80c2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:58.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f559a640 1 -- 172.21.15.107:0/3563079352 >> v1:172.21.15.195:6789/0 conn(0x7f46f80c5c60 legacy=0x7f46f8072680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:58.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f559a640 1 -- 172.21.15.107:0/3563079352 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f80c7ce0 con 0x7f46f8157e10 2023-12-04T05:39:58.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f559a640 1 -- 172.21.15.107:0/3563079352 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f46e8001a00 con 0x7f46f8157e10 2023-12-04T05:39:58.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f559a640 1 -- 172.21.15.107:0/3563079352 >> v1:172.21.15.107:6789/0 conn(0x7f46f8157e10 legacy=0x7f46f81626b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:58.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f559a640 1 --2- 172.21.15.107:0/3563079352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e4003980 0x7f46e4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:58.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f559a640 1 --2- 172.21.15.107:0/3563079352 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f46e40047d0 0x7f46e4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:58.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f559a640 1 --2- 172.21.15.107:0/3563079352 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f46e4005190 0x7f46e400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:58.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f559a640 1 -- 172.21.15.107:0/3563079352 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f80c6b50 con 0x7f46e4003980 2023-12-04T05:39:58.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f559a640 1 -- 172.21.15.107:0/3563079352 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e400c030 con 0x7f46e40047d0 2023-12-04T05:39:58.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f559a640 1 -- 172.21.15.107:0/3563079352 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e400c1f0 con 0x7f46e4005190 2023-12-04T05:39:58.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f659c640 1 --2- 172.21.15.107:0/3563079352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e4003980 0x7f46e4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:58.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f659c640 1 -- 172.21.15.107:0/3563079352 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f46e4005190 msgr2=0x7f46e400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:58.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f659c640 1 --2- 172.21.15.107:0/3563079352 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f46e4005190 0x7f46e400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:58.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f659c640 1 -- 172.21.15.107:0/3563079352 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f46e40047d0 msgr2=0x7f46e4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:58.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f5d9b640 1 --2- 172.21.15.107:0/3563079352 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f46e40047d0 0x7f46e4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:58.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f659c640 1 --2- 172.21.15.107:0/3563079352 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f46e40047d0 0x7f46e4004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:58.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f6d9d640 1 -- 172.21.15.107:0/3563079352 reap_dead start 2023-12-04T05:39:58.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.843+0000 7f46f659c640 1 -- 172.21.15.107:0/3563079352 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e4003530 con 0x7f46e4003980 2023-12-04T05:39:58.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.847+0000 7f46f659c640 1 --2- 172.21.15.107:0/3563079352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e4003980 0x7f46e4003da0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f46f0009ae0 tx=0x7f46f001ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd7a4e3c1b6706a9 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:58.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.847+0000 7f46f559a640 1 -- 172.21.15.107:0/3563079352 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f00a5610 con 0x7f46e4003980 2023-12-04T05:39:58.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.847+0000 7f46f559a640 1 -- 172.21.15.107:0/3563079352 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f00a5770 con 0x7f46e4003980 2023-12-04T05:39:58.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.847+0000 7f46f559a640 1 -- 172.21.15.107:0/3563079352 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f00a5b20 con 0x7f46e4003980 2023-12-04T05:39:58.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.847+0000 7f46fca94640 1 -- 172.21.15.107:0/3563079352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e4003980 msgr2=0x7f46e4003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:58.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.847+0000 7f46fca94640 1 --2- 172.21.15.107:0/3563079352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e4003980 0x7f46e4003da0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f46f0009ae0 tx=0x7f46f001ca20 comp rx=0 tx=0).stop 2023-12-04T05:39:58.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.847+0000 7f46fca94640 1 -- 172.21.15.107:0/3563079352 shutdown_connections 2023-12-04T05:39:58.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.847+0000 7f46fca94640 1 --2- 172.21.15.107:0/3563079352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46e4003980 0x7f46e4003da0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:58.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.847+0000 7f46fca94640 1 -- 172.21.15.107:0/3563079352 >> 172.21.15.107:0/3563079352 conn(0x7f46f806e6f0 msgr2=0x7f46f80c8580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:58.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.847+0000 7f46fca94640 1 -- 172.21.15.107:0/3563079352 shutdown_connections 2023-12-04T05:39:58.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.847+0000 7f46fca94640 1 -- 172.21.15.107:0/3563079352 wait complete. 2023-12-04T05:39:58.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.847+0000 7f46fca94640 1 Processor -- start 2023-12-04T05:39:58.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.847+0000 7f46fca94640 1 -- start start 2023-12-04T05:39:58.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46fca94640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f46e4003980 0x7f46f814d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:58.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46fca94640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46f81504f0 0x7f46f8158f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:58.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46fca94640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f46f8150950 0x7f46f81594b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:58.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46fca94640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f815a340 con 0x7f46f81504f0 2023-12-04T05:39:58.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46fca94640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f815b300 con 0x7f46f8150950 2023-12-04T05:39:58.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46fca94640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46f815abd0 con 0x7f46e4003980 2023-12-04T05:39:58.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46f5d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46f81504f0 0x7f46f8158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:58.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46f5d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46f81504f0 0x7f46f8158f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33704/0 (socket says 172.21.15.107:33704) 2023-12-04T05:39:58.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46f5d9b640 1 -- 172.21.15.107:0/517110365 learned_addr learned my addr 172.21.15.107:0/517110365 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:39:58.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46f659c640 1 --2- 172.21.15.107:0/517110365 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f46e4003980 0x7f46f814d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:58.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46f6d9d640 1 --2- 172.21.15.107:0/517110365 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f46f8150950 0x7f46f81594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:58.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46f5d9b640 1 -- 172.21.15.107:0/517110365 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f46e4003980 msgr2=0x7f46f814d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:58.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46f5d9b640 1 --2- 172.21.15.107:0/517110365 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f46e4003980 0x7f46f814d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:58.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46f5d9b640 1 -- 172.21.15.107:0/517110365 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f46f8150950 msgr2=0x7f46f81594b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:58.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46f5d9b640 1 --2- 172.21.15.107:0/517110365 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f46f8150950 0x7f46f81594b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:58.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46f5d9b640 1 -- 172.21.15.107:0/517110365 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f81501d0 con 0x7f46f81504f0 2023-12-04T05:39:58.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46f5d9b640 1 --2- 172.21.15.107:0/517110365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46f81504f0 0x7f46f8158f70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f46e000d5a0 tx=0x7f46e000da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:58.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46eeffd640 1 -- 172.21.15.107:0/517110365 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e0005480 con 0x7f46f81504f0 2023-12-04T05:39:58.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46eeffd640 1 -- 172.21.15.107:0/517110365 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e0003620 con 0x7f46f81504f0 2023-12-04T05:39:58.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46fca94640 1 -- 172.21.15.107:0/517110365 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46f814f930 con 0x7f46f81504f0 2023-12-04T05:39:58.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46eeffd640 1 -- 172.21.15.107:0/517110365 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e0003990 con 0x7f46f81504f0 2023-12-04T05:39:58.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46fca94640 1 -- 172.21.15.107:0/517110365 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46f814fef0 con 0x7f46f81504f0 2023-12-04T05:39:58.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46fca94640 1 -- 172.21.15.107:0/517110365 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46f8067ec0 con 0x7f46f81504f0 2023-12-04T05:39:58.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46eeffd640 1 -- 172.21.15.107:0/517110365 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f46e0003af0 con 0x7f46f81504f0 2023-12-04T05:39:58.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46eeffd640 1 --2- 172.21.15.107:0/517110365 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f46d80668d0 0x7f46d8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:58.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46f659c640 1 --2- 172.21.15.107:0/517110365 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f46d80668d0 0x7f46d8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:58.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46eeffd640 1 -- 172.21.15.107:0/517110365 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f46e0084310 con 0x7f46f81504f0 2023-12-04T05:39:58.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.851+0000 7f46f659c640 1 --2- 172.21.15.107:0/517110365 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f46d80668d0 0x7f46d8068d90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f46f00af000 tx=0x7f46f0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:58.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:58.855+0000 7f46eeffd640 1 -- 172.21.15.107:0/517110365 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f46e0051520 con 0x7f46f81504f0 2023-12-04T05:39:59.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.027+0000 7f46fca94640 1 -- 172.21.15.107:0/517110365 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f46f8068700 con 0x7f46f81504f0 2023-12-04T05:39:59.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.747+0000 7f46eeffd640 1 -- 172.21.15.107:0/517110365 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f46e0054b40 con 0x7f46f81504f0 2023-12-04T05:39:59.750 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 2 and data pool 3 2023-12-04T05:39:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.751+0000 7f46fca94640 1 -- 172.21.15.107:0/517110365 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f46d80668d0 msgr2=0x7f46d8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.751+0000 7f46fca94640 1 --2- 172.21.15.107:0/517110365 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f46d80668d0 0x7f46d8068d90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f46f00af000 tx=0x7f46f0008040 comp rx=0 tx=0).stop 2023-12-04T05:39:59.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.751+0000 7f46fca94640 1 -- 172.21.15.107:0/517110365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46f81504f0 msgr2=0x7f46f8158f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:59.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.751+0000 7f46fca94640 1 --2- 172.21.15.107:0/517110365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46f81504f0 0x7f46f8158f70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f46e000d5a0 tx=0x7f46e000da60 comp rx=0 tx=0).stop 2023-12-04T05:39:59.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.751+0000 7f46fca94640 1 -- 172.21.15.107:0/517110365 shutdown_connections 2023-12-04T05:39:59.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.751+0000 7f46fca94640 1 --2- 172.21.15.107:0/517110365 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f46d80668d0 0x7f46d8068d90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:59.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.751+0000 7f46fca94640 1 --2- 172.21.15.107:0/517110365 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f46f8150950 0x7f46f81594b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:59.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.751+0000 7f46fca94640 1 --2- 172.21.15.107:0/517110365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46f81504f0 0x7f46f8158f70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:59.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.751+0000 7f46fca94640 1 --2- 172.21.15.107:0/517110365 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f46e4003980 0x7f46f814d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:59.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.751+0000 7f46fca94640 1 -- 172.21.15.107:0/517110365 >> 172.21.15.107:0/517110365 conn(0x7f46f806e6f0 msgr2=0x7f46f80c4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:59.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.751+0000 7f46fca94640 1 -- 172.21.15.107:0/517110365 shutdown_connections 2023-12-04T05:39:59.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.751+0000 7f46fca94640 1 -- 172.21.15.107:0/517110365 wait complete. 2023-12-04T05:39:59.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T05:39:59.845 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:39:59.847 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:39:59.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f067ea00640 1 Processor -- start 2023-12-04T05:39:59.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f067ea00640 1 -- start start 2023-12-04T05:39:59.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f067ea00640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f0678154020 0x7f067815a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:59.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f067ea00640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f06780c4c10 0x7f06780c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:59.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f067ea00640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067815ad50 con 0x7f06780d01c0 2023-12-04T05:39:59.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f067ea00640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06780ca230 con 0x7f0678154020 2023-12-04T05:39:59.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f067ea00640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06780c6770 con 0x7f06780c4c10 2023-12-04T05:39:59.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f066f7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f06780d01c0 0x7f0678071ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60260/0 (socket says 172.21.15.107:60260) 2023-12-04T05:39:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f066f7fe640 1 -- 172.21.15.107:0/3539552317 learned_addr learned my addr 172.21.15.107:0/3539552317 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:39:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f067cf9d640 1 --2- 172.21.15.107:0/3539552317 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f06780c4c10 0x7f06780c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f06777fe640 1 -- 172.21.15.107:0/3539552317 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029616270 0 0) 0x7f067815ad50 con 0x7f06780d01c0 2023-12-04T05:39:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f0677fff640 1 --2- 172.21.15.107:0/3539552317 >> v2:172.21.15.195:3300/0 conn(0x7f0678154020 0x7f067815a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f06777fe640 1 -- 172.21.15.107:0/3539552317 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0664003530 con 0x7f06780d01c0 2023-12-04T05:39:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f06777fe640 1 -- 172.21.15.107:0/3539552317 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3150645029 0 0) 0x7f0664003530 con 0x7f06780d01c0 2023-12-04T05:39:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f06777fe640 1 -- 172.21.15.107:0/3539552317 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f06780c4c10 msgr2=0x7f06780c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f06777fe640 1 --2- 172.21.15.107:0/3539552317 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f06780c4c10 0x7f06780c5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f06777fe640 1 -- 172.21.15.107:0/3539552317 >> v2:172.21.15.195:3300/0 conn(0x7f0678154020 msgr2=0x7f067815a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f06777fe640 1 --2- 172.21.15.107:0/3539552317 >> v2:172.21.15.195:3300/0 conn(0x7f0678154020 0x7f067815a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f06777fe640 1 -- 172.21.15.107:0/3539552317 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06780c55a0 con 0x7f06780d01c0 2023-12-04T05:39:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f06777fe640 1 -- 172.21.15.107:0/3539552317 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f06600016a0 con 0x7f06780d01c0 2023-12-04T05:39:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f06777fe640 1 -- 172.21.15.107:0/3539552317 >> v1:172.21.15.107:6789/0 conn(0x7f06780d01c0 legacy=0x7f0678071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:59.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f06777fe640 1 --2- 172.21.15.107:0/3539552317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0664003980 0x7f0664003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:59.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f06777fe640 1 --2- 172.21.15.107:0/3539552317 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f06780c4c10 0x7f06640047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:59.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f06777fe640 1 --2- 172.21.15.107:0/3539552317 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0664004d10 0x7f0664005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:59.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f06777fe640 1 -- 172.21.15.107:0/3539552317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066400ba20 con 0x7f0664003980 2023-12-04T05:39:59.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f06777fe640 1 -- 172.21.15.107:0/3539552317 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066400bb60 con 0x7f0664004d10 2023-12-04T05:39:59.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f06777fe640 1 -- 172.21.15.107:0/3539552317 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f066400bcd0 con 0x7f06780c4c10 2023-12-04T05:39:59.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f0677fff640 1 --2- 172.21.15.107:0/3539552317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0664003980 0x7f0664003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:59.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f066f7fe640 1 --2- 172.21.15.107:0/3539552317 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f06780c4c10 0x7f06640047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:59.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f067cf9d640 1 --2- 172.21.15.107:0/3539552317 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0664004d10 0x7f0664005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:59.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f0677fff640 1 -- 172.21.15.107:0/3539552317 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f06780c4c10 msgr2=0x7f06640047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:59.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f0677fff640 1 --2- 172.21.15.107:0/3539552317 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f06780c4c10 0x7f06640047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:59.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f0677fff640 1 -- 172.21.15.107:0/3539552317 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0664004d10 msgr2=0x7f0664005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:39:59.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f0677fff640 1 --2- 172.21.15.107:0/3539552317 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0664004d10 0x7f0664005190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:59.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f0677fff640 1 -- 172.21.15.107:0/3539552317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0664003530 con 0x7f0664003980 2023-12-04T05:39:59.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.891+0000 7f066f7fe640 1 --2- 172.21.15.107:0/3539552317 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f06780c4c10 0x7f06640047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:39:59.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f0677fff640 1 --2- 172.21.15.107:0/3539552317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0664003980 0x7f0664003da0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f066800ba30 tx=0x7f066801c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=429ecdc5053494e1 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:59.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f06777fe640 1 -- 172.21.15.107:0/3539552317 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06680aa630 con 0x7f0664003980 2023-12-04T05:39:59.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f06777fe640 1 -- 172.21.15.107:0/3539552317 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06680aa790 con 0x7f0664003980 2023-12-04T05:39:59.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f06777fe640 1 -- 172.21.15.107:0/3539552317 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06680aab40 con 0x7f0664003980 2023-12-04T05:39:59.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f067ea00640 1 -- 172.21.15.107:0/3539552317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0664003980 msgr2=0x7f0664003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:59.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f067ea00640 1 --2- 172.21.15.107:0/3539552317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0664003980 0x7f0664003da0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f066800ba30 tx=0x7f066801c6c0 comp rx=0 tx=0).stop 2023-12-04T05:39:59.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f067cf9d640 1 -- 172.21.15.107:0/3539552317 reap_dead start 2023-12-04T05:39:59.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f067ea00640 1 -- 172.21.15.107:0/3539552317 shutdown_connections 2023-12-04T05:39:59.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f067ea00640 1 -- 172.21.15.107:0/3539552317 >> 172.21.15.107:0/3539552317 conn(0x7f067806c6c0 msgr2=0x7f06780cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:39:59.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f067ea00640 1 -- 172.21.15.107:0/3539552317 shutdown_connections 2023-12-04T05:39:59.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f067ea00640 1 -- 172.21.15.107:0/3539552317 wait complete. 2023-12-04T05:39:59.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f067ea00640 1 Processor -- start 2023-12-04T05:39:59.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f067ea00640 1 -- start start 2023-12-04T05:39:59.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f067ea00640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0664003980 0x7f06781496a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:59.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f067ea00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06781f2f60 0x7f06781f33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:59.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f067ea00640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f06780ce5c0 0x7f06780cea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:59.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f067ea00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06781f2830 con 0x7f06781f2f60 2023-12-04T05:39:59.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f067ea00640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06781f2ad0 con 0x7f0664003980 2023-12-04T05:39:59.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f067ea00640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0678155740 con 0x7f06780ce5c0 2023-12-04T05:39:59.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f066f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06781f2f60 0x7f06781f33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:59.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f066f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06781f2f60 0x7f06781f33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33720/0 (socket says 172.21.15.107:33720) 2023-12-04T05:39:59.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f066f7fe640 1 -- 172.21.15.107:0/1767972247 learned_addr learned my addr 172.21.15.107:0/1767972247 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:39:59.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f067cf9d640 1 --2- 172.21.15.107:0/1767972247 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f06780ce5c0 0x7f06780cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:59.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f066f7fe640 1 -- 172.21.15.107:0/1767972247 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f06780ce5c0 msgr2=0x7f06780cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:59.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f066f7fe640 1 --2- 172.21.15.107:0/1767972247 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f06780ce5c0 0x7f06780cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:59.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f066f7fe640 1 -- 172.21.15.107:0/1767972247 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0664003980 msgr2=0x7f06781496a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:39:59.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f066f7fe640 1 --2- 172.21.15.107:0/1767972247 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0664003980 0x7f06781496a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:39:59.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f066f7fe640 1 -- 172.21.15.107:0/1767972247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0678154960 con 0x7f06781f2f60 2023-12-04T05:39:59.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.895+0000 7f066f7fe640 1 --2- 172.21.15.107:0/1767972247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06781f2f60 0x7f06781f33c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0660001020 tx=0x7f0660011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:59.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.899+0000 7f06757fa640 1 -- 172.21.15.107:0/1767972247 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066000f950 con 0x7f06781f2f60 2023-12-04T05:39:59.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.899+0000 7f06757fa640 1 -- 172.21.15.107:0/1767972247 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066000fab0 con 0x7f06781f2f60 2023-12-04T05:39:59.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.899+0000 7f067ea00640 1 -- 172.21.15.107:0/1767972247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0678154680 con 0x7f06781f2f60 2023-12-04T05:39:59.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.899+0000 7f06757fa640 1 -- 172.21.15.107:0/1767972247 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066000fe40 con 0x7f06781f2f60 2023-12-04T05:39:59.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.899+0000 7f067ea00640 1 -- 172.21.15.107:0/1767972247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f067815c280 con 0x7f06781f2f60 2023-12-04T05:39:59.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.899+0000 7f067ea00640 1 -- 172.21.15.107:0/1767972247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0678067870 con 0x7f06781f2f60 2023-12-04T05:39:59.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.899+0000 7f06757fa640 1 -- 172.21.15.107:0/1767972247 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f066001f460 con 0x7f06781f2f60 2023-12-04T05:39:59.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.899+0000 7f06757fa640 1 --2- 172.21.15.107:0/1767972247 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f06580668d0 0x7f0658068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:39:59.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.899+0000 7f06757fa640 1 -- 172.21.15.107:0/1767972247 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0660087470 con 0x7f06781f2f60 2023-12-04T05:39:59.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.899+0000 7f0677fff640 1 --2- 172.21.15.107:0/1767972247 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f06580668d0 0x7f0658068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:39:59.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.899+0000 7f0677fff640 1 --2- 172.21.15.107:0/1767972247 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f06580668d0 0x7f0658068d90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f06680af000 tx=0x7f06680ac040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:39:59.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:39:59.903+0000 7f06757fa640 1 -- 172.21.15.107:0/1767972247 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f06600546b0 con 0x7f06781f2f60 2023-12-04T05:40:00.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.059+0000 7f067ea00640 1 -- 172.21.15.107:0/1767972247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06780b2590 con 0x7f06781f2f60 2023-12-04T05:40:00.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.063+0000 7f06757fa640 1 -- 172.21.15.107:0/1767972247 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16554 (secure 0 0 0) 0x7f0660057cd0 con 0x7f06781f2f60 2023-12-04T05:40:00.064 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:00.064 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":20,"fsid":"603e856b-31ac-475a-8f35-21ab7faadd48","created":"2023-12-04T05:39:34.921690+0000","modified":"2023-12-04T05:39:59.031537+0000","last_up_change":"2023-12-04T05:39:46.235790+0000","last_in_change":"2023-12-04T05:39:37.587639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T05:39:47.567702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T05:39:57.644286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T05:39:58.029897+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ab7111c-0ae8-4d41-b6db-e3743614322c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6817","nonce":768714279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6821","nonce":768714279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6829","nonce":768714279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6825","nonce":768714279}]},"public_addr":"172.21.15.107:6817/768714279","cluster_addr":"172.21.15.107:6821/768714279","heartbeat_back_addr":"172.21.15.107:6829/768714279","heartbeat_front_addr":"172.21.15.107:6825/768714279","state":["exists","up"]},{"osd":1,"uuid":"baef7a1f-8026-4fab-9e70-48465271034e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6801","nonce":2496355981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6803","nonce":2496355981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6807","nonce":2496355981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6805","nonce":2496355981}]},"public_addr":"172.21.15.107:6801/2496355981","cluster_addr":"172.21.15.107:6803/2496355981","heartbeat_back_addr":"172.21.15.107:6807/2496355981","heartbeat_front_addr":"172.21.15.107:6805/2496355981","state":["exists","up"]},{"osd":2,"uuid":"90e8cf24-7b54-484a-9d85-b44a9d3dfeab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6809","nonce":3097298224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6811","nonce":3097298224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6815","nonce":3097298224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6813","nonce":3097298224}]},"public_addr":"172.21.15.107:6809/3097298224","cluster_addr":"172.21.15.107:6811/3097298224","heartbeat_back_addr":"172.21.15.107:6815/3097298224","heartbeat_front_addr":"172.21.15.107:6813/3097298224","state":["exists","up"]},{"osd":3,"uuid":"54658731-2b9a-4d5e-ab13-7c1be7d34e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6820","nonce":3773076671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6824","nonce":3773076671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6831","nonce":3773076671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6828","nonce":3773076671}]},"public_addr":"172.21.15.107:6820/3773076671","cluster_addr":"172.21.15.107:6824/3773076671","heartbeat_back_addr":"172.21.15.107:6831/3773076671","heartbeat_front_addr":"172.21.15.107:6828/3773076671","state":["exists","up"]},{"osd":4,"uuid":"9a9dea13-8afe-485d-aa43-45eda691179b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6801","nonce":2747823542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6803","nonce":2747823542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6807","nonce":2747823542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6805","nonce":2747823542}]},"public_addr":"172.21.15.195:6801/2747823542","cluster_addr":"172.21.15.195:6803/2747823542","heartbeat_back_addr":"172.21.15.195:6807/2747823542","heartbeat_front_addr":"172.21.15.195:6805/2747823542","state":["exists","up"]},{"osd":5,"uuid":"db840d83-5a7d-468a-8aa2-1997945847c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6825","nonce":1484264540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6827","nonce":1484264540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6831","nonce":1484264540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6829","nonce":1484264540}]},"public_addr":"172.21.15.195:6825/1484264540","cluster_addr":"172.21.15.195:6827/1484264540","heartbeat_back_addr":"172.21.15.195:6831/1484264540","heartbeat_front_addr":"172.21.15.195:6829/1484264540","state":["exists","up"]},{"osd":6,"uuid":"38b5f165-bf91-467e-a615-4b4784337689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6811","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6813","nonce":2638471412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6817","nonce":2638471412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6823","nonce":2638471412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6821","nonce":2638471412}]},"public_addr":"172.21.15.195:6813/2638471412","cluster_addr":"172.21.15.195:6817/2638471412","heartbeat_back_addr":"172.21.15.195:6823/2638471412","heartbeat_front_addr":"172.21.15.195:6821/2638471412","state":["exists","up"]},{"osd":7,"uuid":"e5b3c596-494b-4df3-9437-bc2d52168ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6809","nonce":768189614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6812","nonce":768189614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6820","nonce":768189614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6816","nonce":768189614}]},"public_addr":"172.21.15.195:6809/768189614","cluster_addr":"172.21.15.195:6812/768189614","heartbeat_back_addr":"172.21.15.195:6820/768189614","heartbeat_front_addr":"172.21.15.195:6816/768189614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.209077+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.086601+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.221568+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.238524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.059477+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.148422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.079867+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.064916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T05:40:00.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.063+0000 7f067ea00640 1 -- 172.21.15.107:0/1767972247 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f06580668d0 msgr2=0x7f0658068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:00.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.063+0000 7f067ea00640 1 --2- 172.21.15.107:0/1767972247 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f06580668d0 0x7f0658068d90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f06680af000 tx=0x7f06680ac040 comp rx=0 tx=0).stop 2023-12-04T05:40:00.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.063+0000 7f067ea00640 1 -- 172.21.15.107:0/1767972247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06781f2f60 msgr2=0x7f06781f33c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:00.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.063+0000 7f067ea00640 1 --2- 172.21.15.107:0/1767972247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06781f2f60 0x7f06781f33c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0660001020 tx=0x7f0660011c20 comp rx=0 tx=0).stop 2023-12-04T05:40:00.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.063+0000 7f067ea00640 1 -- 172.21.15.107:0/1767972247 shutdown_connections 2023-12-04T05:40:00.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.063+0000 7f067ea00640 1 --2- 172.21.15.107:0/1767972247 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f06580668d0 0x7f0658068d90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:00.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.063+0000 7f067ea00640 1 --2- 172.21.15.107:0/1767972247 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f06780ce5c0 0x7f06780cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:00.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.063+0000 7f067ea00640 1 --2- 172.21.15.107:0/1767972247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06781f2f60 0x7f06781f33c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:00.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.063+0000 7f067ea00640 1 --2- 172.21.15.107:0/1767972247 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0664003980 0x7f06781496a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:00.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.063+0000 7f067ea00640 1 -- 172.21.15.107:0/1767972247 >> 172.21.15.107:0/1767972247 conn(0x7f067806c6c0 msgr2=0x7f06780cb950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:00.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.063+0000 7f067ea00640 1 -- 172.21.15.107:0/1767972247 shutdown_connections 2023-12-04T05:40:00.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.063+0000 7f067ea00640 1 -- 172.21.15.107:0/1767972247 wait complete. 2023-12-04T05:40:00.089 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T05:40:00.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16df192640 1 Processor -- start 2023-12-04T05:40:00.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16df192640 1 -- start start 2023-12-04T05:40:00.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16df192640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f16d80c7880 0x7f16d8071450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:00.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16df192640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d8156be0 con 0x7f16d80c8b60 2023-12-04T05:40:00.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16df192640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d8156e80 con 0x7f16d806e300 2023-12-04T05:40:00.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16df192640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16d8157140 con 0x7f16d80c7880 2023-12-04T05:40:00.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d7fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f16d80c8b60 0x7f16d8070d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60266/0 (socket says 172.21.15.107:60266) 2023-12-04T05:40:00.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d7fff640 1 -- 172.21.15.107:0/1438073437 learned_addr learned my addr 172.21.15.107:0/1438073437 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:00.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16dd72f640 1 --2- 172.21.15.107:0/1438073437 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f16d80c7880 0x7f16d8071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:00.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d77fe640 1 -- 172.21.15.107:0/1438073437 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889873727 0 0) 0x7f16d8156be0 con 0x7f16d80c8b60 2023-12-04T05:40:00.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d77fe640 1 -- 172.21.15.107:0/1438073437 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16c8003530 con 0x7f16d80c8b60 2023-12-04T05:40:00.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d77fe640 1 -- 172.21.15.107:0/1438073437 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3033336004 0 0) 0x7f16c8003530 con 0x7f16d80c8b60 2023-12-04T05:40:00.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d77fe640 1 -- 172.21.15.107:0/1438073437 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f16d80c7880 msgr2=0x7f16d8071450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:00.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d77fe640 1 --2- 172.21.15.107:0/1438073437 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f16d80c7880 0x7f16d8071450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:00.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d77fe640 1 -- 172.21.15.107:0/1438073437 >> v1:172.21.15.195:6789/0 conn(0x7f16d806e300 legacy=0x7f16d80c7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:00.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d77fe640 1 -- 172.21.15.107:0/1438073437 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d8157420 con 0x7f16d80c8b60 2023-12-04T05:40:00.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d77fe640 1 -- 172.21.15.107:0/1438073437 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f16c4001680 con 0x7f16d80c8b60 2023-12-04T05:40:00.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d77fe640 1 -- 172.21.15.107:0/1438073437 >> v1:172.21.15.107:6789/0 conn(0x7f16d80c8b60 legacy=0x7f16d8070d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:00.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d77fe640 1 --2- 172.21.15.107:0/1438073437 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f16d80c7880 0x7f16c80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:00.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d77fe640 1 --2- 172.21.15.107:0/1438073437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16c8003ef0 0x7f16c80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:00.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d77fe640 1 --2- 172.21.15.107:0/1438073437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16c8004d10 0x7f16c8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:00.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d77fe640 1 -- 172.21.15.107:0/1438073437 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d8157140 con 0x7f16c8003ef0 2023-12-04T05:40:00.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d77fe640 1 -- 172.21.15.107:0/1438073437 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c800ffa0 con 0x7f16c8004d10 2023-12-04T05:40:00.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d77fe640 1 -- 172.21.15.107:0/1438073437 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c8010130 con 0x7f16d80c7880 2023-12-04T05:40:00.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d7fff640 1 --2- 172.21.15.107:0/1438073437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16c8003ef0 0x7f16c80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:00.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16dcf2e640 1 --2- 172.21.15.107:0/1438073437 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f16d80c7880 0x7f16c80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:00.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16dd72f640 1 --2- 172.21.15.107:0/1438073437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16c8004d10 0x7f16c8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:00.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d7fff640 1 -- 172.21.15.107:0/1438073437 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f16d80c7880 msgr2=0x7f16c80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:00.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d7fff640 1 --2- 172.21.15.107:0/1438073437 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f16d80c7880 0x7f16c80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:00.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d7fff640 1 -- 172.21.15.107:0/1438073437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16c8004d10 msgr2=0x7f16c8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:00.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d7fff640 1 --2- 172.21.15.107:0/1438073437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16c8004d10 0x7f16c8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:00.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16d7fff640 1 -- 172.21.15.107:0/1438073437 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c8003530 con 0x7f16c8003ef0 2023-12-04T05:40:00.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.251+0000 7f16dd72f640 1 --2- 172.21.15.107:0/1438073437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16c8004d10 0x7f16c8005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:40:00.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.255+0000 7f16d7fff640 1 --2- 172.21.15.107:0/1438073437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16c8003ef0 0x7f16c80047d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f16c400c110 tx=0x7f16c401cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a454a300d24942 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:00.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.255+0000 7f16d77fe640 1 -- 172.21.15.107:0/1438073437 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c4005ae0 con 0x7f16c8003ef0 2023-12-04T05:40:00.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.255+0000 7f16d77fe640 1 -- 172.21.15.107:0/1438073437 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16c4005c40 con 0x7f16c8003ef0 2023-12-04T05:40:00.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.255+0000 7f16d77fe640 1 -- 172.21.15.107:0/1438073437 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c40a5aa0 con 0x7f16c8003ef0 2023-12-04T05:40:00.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.255+0000 7f16df192640 1 -- 172.21.15.107:0/1438073437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16c8003ef0 msgr2=0x7f16c80047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:00.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.255+0000 7f16df192640 1 --2- 172.21.15.107:0/1438073437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16c8003ef0 0x7f16c80047d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f16c400c110 tx=0x7f16c401cb60 comp rx=0 tx=0).stop 2023-12-04T05:40:00.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.255+0000 7f16dd72f640 1 -- 172.21.15.107:0/1438073437 reap_dead start 2023-12-04T05:40:00.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.255+0000 7f16df192640 1 -- 172.21.15.107:0/1438073437 shutdown_connections 2023-12-04T05:40:00.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.255+0000 7f16df192640 1 -- 172.21.15.107:0/1438073437 >> 172.21.15.107:0/1438073437 conn(0x7f16d806b6c0 msgr2=0x7f16d806bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:00.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.255+0000 7f16df192640 1 -- 172.21.15.107:0/1438073437 shutdown_connections 2023-12-04T05:40:00.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.255+0000 7f16df192640 1 -- 172.21.15.107:0/1438073437 wait complete. 2023-12-04T05:40:00.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.255+0000 7f16df192640 1 Processor -- start 2023-12-04T05:40:00.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.255+0000 7f16df192640 1 -- start start 2023-12-04T05:40:00.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16df192640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16c8003ef0 0x7f16d81eebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:00.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16df192640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f16d80c5790 0x7f16d8155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:00.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16df192640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16d81ef120 0x7f16d8156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:00.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16df192640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d81554d0 con 0x7f16d81ef120 2023-12-04T05:40:00.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16df192640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d80c5520 con 0x7f16c8003ef0 2023-12-04T05:40:00.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16df192640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16d80c5d00 con 0x7f16d80c5790 2023-12-04T05:40:00.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16dd72f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16d81ef120 0x7f16d8156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:00.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16dd72f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16d81ef120 0x7f16d8156130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33750/0 (socket says 172.21.15.107:33750) 2023-12-04T05:40:00.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16dd72f640 1 -- 172.21.15.107:0/318795228 learned_addr learned my addr 172.21.15.107:0/318795228 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:00.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16d7fff640 1 --2- 172.21.15.107:0/318795228 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f16d80c5790 0x7f16d8155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:00.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16dcf2e640 1 --2- 172.21.15.107:0/318795228 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16c8003ef0 0x7f16d81eebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:00.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16dd72f640 1 -- 172.21.15.107:0/318795228 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f16d80c5790 msgr2=0x7f16d8155bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:00.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16dd72f640 1 --2- 172.21.15.107:0/318795228 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f16d80c5790 0x7f16d8155bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:00.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16dd72f640 1 -- 172.21.15.107:0/318795228 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16c8003ef0 msgr2=0x7f16d81eebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:00.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16dd72f640 1 --2- 172.21.15.107:0/318795228 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16c8003ef0 0x7f16d81eebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:00.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16dd72f640 1 -- 172.21.15.107:0/318795228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d80c5fc0 con 0x7f16d81ef120 2023-12-04T05:40:00.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16dd72f640 1 --2- 172.21.15.107:0/318795228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16d81ef120 0x7f16d8156130 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f16d8070690 tx=0x7f16d0010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:00.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16d57fa640 1 -- 172.21.15.107:0/318795228 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d000e050 con 0x7f16d81ef120 2023-12-04T05:40:00.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16d57fa640 1 -- 172.21.15.107:0/318795228 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d000f040 con 0x7f16d81ef120 2023-12-04T05:40:00.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16d57fa640 1 -- 172.21.15.107:0/318795228 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d0003be0 con 0x7f16d81ef120 2023-12-04T05:40:00.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16df192640 1 -- 172.21.15.107:0/318795228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16d81ef660 con 0x7f16d81ef120 2023-12-04T05:40:00.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16df192640 1 -- 172.21.15.107:0/318795228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16d81df700 con 0x7f16d81ef120 2023-12-04T05:40:00.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16df192640 1 -- 172.21.15.107:0/318795228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16d8155910 con 0x7f16d81ef120 2023-12-04T05:40:00.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.259+0000 7f16d57fa640 1 -- 172.21.15.107:0/318795228 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f16d0003d40 con 0x7f16d81ef120 2023-12-04T05:40:00.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.263+0000 7f16d57fa640 1 --2- 172.21.15.107:0/318795228 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f16bc0668d0 0x7f16bc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:00.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.263+0000 7f16d57fa640 1 -- 172.21.15.107:0/318795228 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f16d0085570 con 0x7f16d81ef120 2023-12-04T05:40:00.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.263+0000 7f16dcf2e640 1 --2- 172.21.15.107:0/318795228 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f16bc0668d0 0x7f16bc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:00.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.263+0000 7f16dcf2e640 1 --2- 172.21.15.107:0/318795228 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f16bc0668d0 0x7f16bc068d90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f16cc000c10 tx=0x7f16cc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:00.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.263+0000 7f16d57fa640 1 -- 172.21.15.107:0/318795228 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f16d0052920 con 0x7f16d81ef120 2023-12-04T05:40:00.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.419+0000 7f16df192640 1 -- 172.21.15.107:0/318795228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16d8155910 con 0x7f16d81ef120 2023-12-04T05:40:00.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.423+0000 7f16d57fa640 1 -- 172.21.15.107:0/318795228 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16554 (secure 0 0 0) 0x7f16d8155910 con 0x7f16d81ef120 2023-12-04T05:40:00.425 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:00.425 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":20,"fsid":"603e856b-31ac-475a-8f35-21ab7faadd48","created":"2023-12-04T05:39:34.921690+0000","modified":"2023-12-04T05:39:59.031537+0000","last_up_change":"2023-12-04T05:39:46.235790+0000","last_in_change":"2023-12-04T05:39:37.587639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T05:39:47.567702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T05:39:57.644286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T05:39:58.029897+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ab7111c-0ae8-4d41-b6db-e3743614322c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6817","nonce":768714279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6821","nonce":768714279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6829","nonce":768714279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6825","nonce":768714279}]},"public_addr":"172.21.15.107:6817/768714279","cluster_addr":"172.21.15.107:6821/768714279","heartbeat_back_addr":"172.21.15.107:6829/768714279","heartbeat_front_addr":"172.21.15.107:6825/768714279","state":["exists","up"]},{"osd":1,"uuid":"baef7a1f-8026-4fab-9e70-48465271034e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6801","nonce":2496355981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6803","nonce":2496355981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6807","nonce":2496355981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6805","nonce":2496355981}]},"public_addr":"172.21.15.107:6801/2496355981","cluster_addr":"172.21.15.107:6803/2496355981","heartbeat_back_addr":"172.21.15.107:6807/2496355981","heartbeat_front_addr":"172.21.15.107:6805/2496355981","state":["exists","up"]},{"osd":2,"uuid":"90e8cf24-7b54-484a-9d85-b44a9d3dfeab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6809","nonce":3097298224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6811","nonce":3097298224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6815","nonce":3097298224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6813","nonce":3097298224}]},"public_addr":"172.21.15.107:6809/3097298224","cluster_addr":"172.21.15.107:6811/3097298224","heartbeat_back_addr":"172.21.15.107:6815/3097298224","heartbeat_front_addr":"172.21.15.107:6813/3097298224","state":["exists","up"]},{"osd":3,"uuid":"54658731-2b9a-4d5e-ab13-7c1be7d34e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6820","nonce":3773076671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6824","nonce":3773076671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6831","nonce":3773076671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6828","nonce":3773076671}]},"public_addr":"172.21.15.107:6820/3773076671","cluster_addr":"172.21.15.107:6824/3773076671","heartbeat_back_addr":"172.21.15.107:6831/3773076671","heartbeat_front_addr":"172.21.15.107:6828/3773076671","state":["exists","up"]},{"osd":4,"uuid":"9a9dea13-8afe-485d-aa43-45eda691179b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6801","nonce":2747823542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6803","nonce":2747823542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6807","nonce":2747823542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6805","nonce":2747823542}]},"public_addr":"172.21.15.195:6801/2747823542","cluster_addr":"172.21.15.195:6803/2747823542","heartbeat_back_addr":"172.21.15.195:6807/2747823542","heartbeat_front_addr":"172.21.15.195:6805/2747823542","state":["exists","up"]},{"osd":5,"uuid":"db840d83-5a7d-468a-8aa2-1997945847c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6825","nonce":1484264540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6827","nonce":1484264540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6831","nonce":1484264540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6829","nonce":1484264540}]},"public_addr":"172.21.15.195:6825/1484264540","cluster_addr":"172.21.15.195:6827/1484264540","heartbeat_back_addr":"172.21.15.195:6831/1484264540","heartbeat_front_addr":"172.21.15.195:6829/1484264540","state":["exists","up"]},{"osd":6,"uuid":"38b5f165-bf91-467e-a615-4b4784337689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6811","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6813","nonce":2638471412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6817","nonce":2638471412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6823","nonce":2638471412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6821","nonce":2638471412}]},"public_addr":"172.21.15.195:6813/2638471412","cluster_addr":"172.21.15.195:6817/2638471412","heartbeat_back_addr":"172.21.15.195:6823/2638471412","heartbeat_front_addr":"172.21.15.195:6821/2638471412","state":["exists","up"]},{"osd":7,"uuid":"e5b3c596-494b-4df3-9437-bc2d52168ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6809","nonce":768189614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6812","nonce":768189614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6820","nonce":768189614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6816","nonce":768189614}]},"public_addr":"172.21.15.195:6809/768189614","cluster_addr":"172.21.15.195:6812/768189614","heartbeat_back_addr":"172.21.15.195:6820/768189614","heartbeat_front_addr":"172.21.15.195:6816/768189614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.209077+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.086601+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.221568+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.238524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.059477+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.148422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.079867+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.064916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T05:40:00.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.423+0000 7f16df192640 1 -- 172.21.15.107:0/318795228 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f16bc0668d0 msgr2=0x7f16bc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:00.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.423+0000 7f16df192640 1 --2- 172.21.15.107:0/318795228 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f16bc0668d0 0x7f16bc068d90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f16cc000c10 tx=0x7f16cc009040 comp rx=0 tx=0).stop 2023-12-04T05:40:00.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.423+0000 7f16df192640 1 -- 172.21.15.107:0/318795228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16d81ef120 msgr2=0x7f16d8156130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:00.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.423+0000 7f16df192640 1 --2- 172.21.15.107:0/318795228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16d81ef120 0x7f16d8156130 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f16d8070690 tx=0x7f16d0010590 comp rx=0 tx=0).stop 2023-12-04T05:40:00.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.423+0000 7f16df192640 1 -- 172.21.15.107:0/318795228 shutdown_connections 2023-12-04T05:40:00.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.423+0000 7f16df192640 1 --2- 172.21.15.107:0/318795228 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f16bc0668d0 0x7f16bc068d90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:00.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.423+0000 7f16df192640 1 --2- 172.21.15.107:0/318795228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16d81ef120 0x7f16d8156130 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:00.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.423+0000 7f16df192640 1 --2- 172.21.15.107:0/318795228 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f16d80c5790 0x7f16d8155bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:00.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.423+0000 7f16df192640 1 --2- 172.21.15.107:0/318795228 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f16c8003ef0 0x7f16d81eebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:00.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.423+0000 7f16df192640 1 -- 172.21.15.107:0/318795228 >> 172.21.15.107:0/318795228 conn(0x7f16d806b6c0 msgr2=0x7f16d814ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:00.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.423+0000 7f16df192640 1 -- 172.21.15.107:0/318795228 shutdown_connections 2023-12-04T05:40:00.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.423+0000 7f16df192640 1 -- 172.21.15.107:0/318795228 wait complete. 2023-12-04T05:40:00.445 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-04T05:40:00.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.563+0000 7ff1e5022640 1 Processor -- start 2023-12-04T05:40:00.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.563+0000 7ff1e5022640 1 -- start start 2023-12-04T05:40:00.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.563+0000 7ff1e5022640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff1e0151f80 0x7ff1e0071030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:00.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.563+0000 7ff1e5022640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1e006c2b0 0x7ff1e00c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:00.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.563+0000 7ff1e5022640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e0152360 con 0x7ff1e00c60e0 2023-12-04T05:40:00.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.563+0000 7ff1e5022640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1e006acf0 con 0x7ff1e0151f80 2023-12-04T05:40:00.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.563+0000 7ff1e5022640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e0157c90 con 0x7ff1e006c2b0 2023-12-04T05:40:00.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.563+0000 7ff1ded9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff1e0151f80 0x7ff1e0071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:00.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.563+0000 7ff1df59e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff1e00c60e0 0x7ff1e006e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60282/0 (socket says 172.21.15.107:60282) 2023-12-04T05:40:00.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.563+0000 7ff1df59e640 1 -- 172.21.15.107:0/1929701482 learned_addr learned my addr 172.21.15.107:0/1929701482 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:00.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.563+0000 7ff1de59c640 1 --2- 172.21.15.107:0/1929701482 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1e006c2b0 0x7ff1e00c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:00.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.563+0000 7ff1ded9d640 1 -- 172.21.15.107:0/1929701482 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1e006c2b0 msgr2=0x7ff1e00c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:00.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.563+0000 7ff1ded9d640 1 --2- 172.21.15.107:0/1929701482 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1e006c2b0 0x7ff1e00c5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:00.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.563+0000 7ff1ded9d640 1 -- 172.21.15.107:0/1929701482 >> v1:172.21.15.107:6789/0 conn(0x7ff1e00c60e0 legacy=0x7ff1e006e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:00.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.563+0000 7ff1ded9d640 1 -- 172.21.15.107:0/1929701482 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e0157f30 con 0x7ff1e0151f80 2023-12-04T05:40:00.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.563+0000 7ff1ddd9b640 1 -- 172.21.15.107:0/1929701482 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203673239 0 0) 0x7ff1e0152360 con 0x7ff1e00c60e0 2023-12-04T05:40:00.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1ded9d640 1 --2- 172.21.15.107:0/1929701482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1e0151f80 0x7ff1e0071030 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff1d00097c0 tx=0x7ff1d001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=597dcac0bd1ec12f server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:00.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1ddd9b640 1 -- 172.21.15.107:0/1929701482 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d00a8970 con 0x7ff1e0151f80 2023-12-04T05:40:00.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1ddd9b640 1 -- 172.21.15.107:0/1929701482 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d00a8ad0 con 0x7ff1e0151f80 2023-12-04T05:40:00.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1ddd9b640 1 -- 172.21.15.107:0/1929701482 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d00a8e60 con 0x7ff1e0151f80 2023-12-04T05:40:00.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1e5022640 1 -- 172.21.15.107:0/1929701482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1e0151f80 msgr2=0x7ff1e0071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:00.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1e5022640 1 --2- 172.21.15.107:0/1929701482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1e0151f80 0x7ff1e0071030 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff1d00097c0 tx=0x7ff1d001ec70 comp rx=0 tx=0).stop 2023-12-04T05:40:00.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1e5022640 1 -- 172.21.15.107:0/1929701482 shutdown_connections 2023-12-04T05:40:00.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1e5022640 1 --2- 172.21.15.107:0/1929701482 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1e006c2b0 0x7ff1e00c5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:00.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1e5022640 1 --2- 172.21.15.107:0/1929701482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1e0151f80 0x7ff1e0071030 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:00.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1e5022640 1 -- 172.21.15.107:0/1929701482 >> 172.21.15.107:0/1929701482 conn(0x7ff1e00cc490 msgr2=0x7ff1e00cc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:00.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1e5022640 1 -- 172.21.15.107:0/1929701482 shutdown_connections 2023-12-04T05:40:00.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1e5022640 1 -- 172.21.15.107:0/1929701482 wait complete. 2023-12-04T05:40:00.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1e5022640 1 Processor -- start 2023-12-04T05:40:00.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1e5022640 1 -- start start 2023-12-04T05:40:00.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1e5022640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1e006c2b0 0x7ff1e01f0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:00.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1e5022640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1e00c60e0 0x7ff1e01effb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:00.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1e5022640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1e0151f80 0x7ff1e01f04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:00.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1e5022640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e01efb90 con 0x7ff1e0151f80 2023-12-04T05:40:00.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1e5022640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e01e1a40 con 0x7ff1e006c2b0 2023-12-04T05:40:00.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1e5022640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e01ef560 con 0x7ff1e00c60e0 2023-12-04T05:40:00.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1df59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1e0151f80 0x7ff1e01f04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:00.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1df59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1e0151f80 0x7ff1e01f04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33762/0 (socket says 172.21.15.107:33762) 2023-12-04T05:40:00.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1df59e640 1 -- 172.21.15.107:0/1102112619 learned_addr learned my addr 172.21.15.107:0/1102112619 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:00.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1ded9d640 1 --2- 172.21.15.107:0/1102112619 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1e006c2b0 0x7ff1e01f0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:00.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1de59c640 1 --2- 172.21.15.107:0/1102112619 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1e00c60e0 0x7ff1e01effb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:00.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1df59e640 1 -- 172.21.15.107:0/1102112619 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1e00c60e0 msgr2=0x7ff1e01effb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:00.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1df59e640 1 --2- 172.21.15.107:0/1102112619 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1e00c60e0 0x7ff1e01effb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:00.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1df59e640 1 -- 172.21.15.107:0/1102112619 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1e006c2b0 msgr2=0x7ff1e01f0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:00.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1df59e640 1 --2- 172.21.15.107:0/1102112619 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1e006c2b0 0x7ff1e01f0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:00.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1df59e640 1 -- 172.21.15.107:0/1102112619 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d0009420 con 0x7ff1e0151f80 2023-12-04T05:40:00.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1ded9d640 1 --2- 172.21.15.107:0/1102112619 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1e006c2b0 0x7ff1e01f0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:00.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1df59e640 1 --2- 172.21.15.107:0/1102112619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1e0151f80 0x7ff1e01f04f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800c810 tx=0x7ff1d800ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:00.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.567+0000 7ff1c77fe640 1 -- 172.21.15.107:0/1102112619 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d800a640 con 0x7ff1e0151f80 2023-12-04T05:40:00.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.571+0000 7ff1c77fe640 1 -- 172.21.15.107:0/1102112619 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d800a7c0 con 0x7ff1e0151f80 2023-12-04T05:40:00.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.571+0000 7ff1c77fe640 1 -- 172.21.15.107:0/1102112619 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d800ab70 con 0x7ff1e0151f80 2023-12-04T05:40:00.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.571+0000 7ff1e5022640 1 -- 172.21.15.107:0/1102112619 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e014cef0 con 0x7ff1e0151f80 2023-12-04T05:40:00.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.571+0000 7ff1e5022640 1 -- 172.21.15.107:0/1102112619 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e01f1a60 con 0x7ff1e0151f80 2023-12-04T05:40:00.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.571+0000 7ff1e5022640 1 -- 172.21.15.107:0/1102112619 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1e01efcd0 con 0x7ff1e0151f80 2023-12-04T05:40:00.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.571+0000 7ff1c77fe640 1 -- 172.21.15.107:0/1102112619 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff1d800acd0 con 0x7ff1e0151f80 2023-12-04T05:40:00.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.571+0000 7ff1c77fe640 1 --2- 172.21.15.107:0/1102112619 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff1cc0668d0 0x7ff1cc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:00.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.571+0000 7ff1c77fe640 1 -- 172.21.15.107:0/1102112619 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1d8083970 con 0x7ff1e0151f80 2023-12-04T05:40:00.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.571+0000 7ff1ded9d640 1 --2- 172.21.15.107:0/1102112619 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff1cc0668d0 0x7ff1cc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:00.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.571+0000 7ff1ded9d640 1 --2- 172.21.15.107:0/1102112619 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff1cc0668d0 0x7ff1cc068d90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff1d00b2000 tx=0x7ff1d00aa070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:00.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.575+0000 7ff1c77fe640 1 -- 172.21.15.107:0/1102112619 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff1e01efcd0 con 0x7ff1e0151f80 2023-12-04T05:40:00.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:00.767+0000 7ff1e5022640 1 -- 172.21.15.107:0/1102112619 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff1e01efcd0 con 0x7ff1e0151f80 2023-12-04T05:40:01.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.751+0000 7ff1c77fe640 1 -- 172.21.15.107:0/1102112619 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff1e01efcd0 con 0x7ff1e0151f80 2023-12-04T05:40:01.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.751+0000 7ff1e5022640 1 -- 172.21.15.107:0/1102112619 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff1cc0668d0 msgr2=0x7ff1cc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:01.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.751+0000 7ff1e5022640 1 --2- 172.21.15.107:0/1102112619 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff1cc0668d0 0x7ff1cc068d90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff1d00b2000 tx=0x7ff1d00aa070 comp rx=0 tx=0).stop 2023-12-04T05:40:01.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.751+0000 7ff1e5022640 1 -- 172.21.15.107:0/1102112619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1e0151f80 msgr2=0x7ff1e01f04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:01.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.751+0000 7ff1e5022640 1 --2- 172.21.15.107:0/1102112619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1e0151f80 0x7ff1e01f04f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800c810 tx=0x7ff1d800ccd0 comp rx=0 tx=0).stop 2023-12-04T05:40:01.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.751+0000 7ff1e5022640 1 -- 172.21.15.107:0/1102112619 shutdown_connections 2023-12-04T05:40:01.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.751+0000 7ff1e5022640 1 --2- 172.21.15.107:0/1102112619 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff1cc0668d0 0x7ff1cc068d90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:01.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.751+0000 7ff1e5022640 1 --2- 172.21.15.107:0/1102112619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1e0151f80 0x7ff1e01f04f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:01.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.751+0000 7ff1e5022640 1 --2- 172.21.15.107:0/1102112619 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1e00c60e0 0x7ff1e01effb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:01.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.751+0000 7ff1e5022640 1 --2- 172.21.15.107:0/1102112619 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1e006c2b0 0x7ff1e01f0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:01.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.755+0000 7ff1e5022640 1 -- 172.21.15.107:0/1102112619 >> 172.21.15.107:0/1102112619 conn(0x7ff1e00cc490 msgr2=0x7ff1e006b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:01.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.755+0000 7ff1e5022640 1 -- 172.21.15.107:0/1102112619 shutdown_connections 2023-12-04T05:40:01.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.755+0000 7ff1e5022640 1 -- 172.21.15.107:0/1102112619 wait complete. 2023-12-04T05:40:01.777 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300} 2023-12-04T05:40:01.777 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2023-12-04T05:40:01.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.899+0000 7f484a73f640 1 Processor -- start 2023-12-04T05:40:01.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.899+0000 7f484a73f640 1 -- start start 2023-12-04T05:40:01.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f484a73f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48440cb030 0x7f48440cb460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:01.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f484a73f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f484415e310 con 0x7f4844153e90 2023-12-04T05:40:01.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f484a73f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48440cd1d0 con 0x7f48440ccd90 2023-12-04T05:40:01.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f484a73f640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48440c6100 con 0x7f48440cb030 2023-12-04T05:40:01.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f4843fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4844153e90 0x7f484415a810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60296/0 (socket says 172.21.15.107:60296) 2023-12-04T05:40:01.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f4843fff640 1 -- 172.21.15.107:0/1548342567 learned_addr learned my addr 172.21.15.107:0/1548342567 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f48437fe640 1 -- 172.21.15.107:0/1548342567 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960942926 0 0) 0x7f484415e310 con 0x7f4844153e90 2023-12-04T05:40:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f48437fe640 1 -- 172.21.15.107:0/1548342567 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4830003530 con 0x7f4844153e90 2023-12-04T05:40:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f4848cdc640 1 --2- 172.21.15.107:0/1548342567 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48440cb030 0x7f48440cb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f48437fe640 1 -- 172.21.15.107:0/1548342567 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 650921211 0 0) 0x7f4830003530 con 0x7f4844153e90 2023-12-04T05:40:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f48437fe640 1 -- 172.21.15.107:0/1548342567 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48440cb030 msgr2=0x7f48440cb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f48437fe640 1 --2- 172.21.15.107:0/1548342567 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48440cb030 0x7f48440cb460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f48437fe640 1 -- 172.21.15.107:0/1548342567 >> v1:172.21.15.195:6789/0 conn(0x7f48440ccd90 legacy=0x7f484406be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:01.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f48437fe640 1 -- 172.21.15.107:0/1548342567 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4844154270 con 0x7f4844153e90 2023-12-04T05:40:01.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f48437fe640 1 -- 172.21.15.107:0/1548342567 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f48340016a0 con 0x7f4844153e90 2023-12-04T05:40:01.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f48437fe640 1 -- 172.21.15.107:0/1548342567 >> v1:172.21.15.107:6789/0 conn(0x7f4844153e90 legacy=0x7f484415a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:01.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f48437fe640 1 --2- 172.21.15.107:0/1548342567 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48300039b0 0x7f4830003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:01.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f48437fe640 1 --2- 172.21.15.107:0/1548342567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48300047d0 0x7f4830004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:01.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f48437fe640 1 --2- 172.21.15.107:0/1548342567 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48440cb030 0x7f48300051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:01.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f48437fe640 1 -- 172.21.15.107:0/1548342567 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483000bb50 con 0x7f48300047d0 2023-12-04T05:40:01.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f48437fe640 1 -- 172.21.15.107:0/1548342567 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483000bcc0 con 0x7f48300039b0 2023-12-04T05:40:01.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f48437fe640 1 -- 172.21.15.107:0/1548342567 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483000be80 con 0x7f48440cb030 2023-12-04T05:40:01.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f483b7fe640 1 --2- 172.21.15.107:0/1548342567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48300047d0 0x7f4830004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:01.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f4843fff640 1 --2- 172.21.15.107:0/1548342567 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48300039b0 0x7f4830003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:01.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f4848cdc640 1 --2- 172.21.15.107:0/1548342567 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48440cb030 0x7f48300051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:01.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f483b7fe640 1 -- 172.21.15.107:0/1548342567 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48440cb030 msgr2=0x7f48300051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:01.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f483b7fe640 1 --2- 172.21.15.107:0/1548342567 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48440cb030 0x7f48300051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:01.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f483b7fe640 1 -- 172.21.15.107:0/1548342567 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48300039b0 msgr2=0x7f4830003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:01.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f483b7fe640 1 --2- 172.21.15.107:0/1548342567 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48300039b0 0x7f4830003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:01.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f483b7fe640 1 -- 172.21.15.107:0/1548342567 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4830003530 con 0x7f48300047d0 2023-12-04T05:40:01.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.903+0000 7f483b7fe640 1 --2- 172.21.15.107:0/1548342567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48300047d0 0x7f4830004c50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f482c0017b0 tx=0x7f482c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f17e8bed903b7d1 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:01.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f48437fe640 1 -- 172.21.15.107:0/1548342567 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c0052d0 con 0x7f48300047d0 2023-12-04T05:40:01.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f48437fe640 1 -- 172.21.15.107:0/1548342567 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482c0a7050 con 0x7f48300047d0 2023-12-04T05:40:01.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f48437fe640 1 -- 172.21.15.107:0/1548342567 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c003650 con 0x7f48300047d0 2023-12-04T05:40:01.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f484a73f640 1 -- 172.21.15.107:0/1548342567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48300047d0 msgr2=0x7f4830004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:01.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f484a73f640 1 --2- 172.21.15.107:0/1548342567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48300047d0 0x7f4830004c50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f482c0017b0 tx=0x7f482c01c3e0 comp rx=0 tx=0).stop 2023-12-04T05:40:01.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f4848cdc640 1 -- 172.21.15.107:0/1548342567 reap_dead start 2023-12-04T05:40:01.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f484a73f640 1 -- 172.21.15.107:0/1548342567 shutdown_connections 2023-12-04T05:40:01.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f484a73f640 1 -- 172.21.15.107:0/1548342567 >> 172.21.15.107:0/1548342567 conn(0x7f484406b790 msgr2=0x7f48440663d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:01.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f484a73f640 1 -- 172.21.15.107:0/1548342567 shutdown_connections 2023-12-04T05:40:01.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f484a73f640 1 -- 172.21.15.107:0/1548342567 wait complete. 2023-12-04T05:40:01.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f484a73f640 1 Processor -- start 2023-12-04T05:40:01.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f484a73f640 1 -- start start 2023-12-04T05:40:01.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f484a73f640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48300047d0 0x7f48441e5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:01.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f484a73f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48441f06f0 0x7f48440b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:01.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f484a73f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48441e5360 0x7f48440ce520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:01.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f484a73f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48441e4d10 con 0x7f48441f06f0 2023-12-04T05:40:01.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f484a73f640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48441e5ff0 con 0x7f48300047d0 2023-12-04T05:40:01.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f484a73f640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48441f0be0 con 0x7f48441e5360 2023-12-04T05:40:01.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f483b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48441f06f0 0x7f48440b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:01.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f483b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48441f06f0 0x7f48440b2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33790/0 (socket says 172.21.15.107:33790) 2023-12-04T05:40:01.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f483b7fe640 1 -- 172.21.15.107:0/1921270470 learned_addr learned my addr 172.21.15.107:0/1921270470 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:01.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f4843fff640 1 --2- 172.21.15.107:0/1921270470 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48300047d0 0x7f48441e5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:01.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f4848cdc640 1 --2- 172.21.15.107:0/1921270470 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48441e5360 0x7f48440ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:01.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f483b7fe640 1 -- 172.21.15.107:0/1921270470 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48441e5360 msgr2=0x7f48440ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:01.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f483b7fe640 1 --2- 172.21.15.107:0/1921270470 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48441e5360 0x7f48440ce520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:01.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f483b7fe640 1 -- 172.21.15.107:0/1921270470 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48300047d0 msgr2=0x7f48441e5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:01.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f483b7fe640 1 --2- 172.21.15.107:0/1921270470 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48300047d0 0x7f48441e5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:01.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f483b7fe640 1 -- 172.21.15.107:0/1921270470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48441e40d0 con 0x7f48441f06f0 2023-12-04T05:40:01.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f4843fff640 1 --2- 172.21.15.107:0/1921270470 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48300047d0 0x7f48441e5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:01.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f483b7fe640 1 --2- 172.21.15.107:0/1921270470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48441f06f0 0x7f48440b2e50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f482c005040 tx=0x7f4830005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:01.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f48417fa640 1 -- 172.21.15.107:0/1921270470 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c007a90 con 0x7f48441f06f0 2023-12-04T05:40:01.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f48417fa640 1 -- 172.21.15.107:0/1921270470 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482c000ee0 con 0x7f48441f06f0 2023-12-04T05:40:01.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f48417fa640 1 -- 172.21.15.107:0/1921270470 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c007260 con 0x7f48441f06f0 2023-12-04T05:40:01.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f484a73f640 1 -- 172.21.15.107:0/1921270470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48441e4370 con 0x7f48441f06f0 2023-12-04T05:40:01.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.907+0000 7f484a73f640 1 -- 172.21.15.107:0/1921270470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48441e4930 con 0x7f48441f06f0 2023-12-04T05:40:01.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.911+0000 7f484a73f640 1 -- 172.21.15.107:0/1921270470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48440c59f0 con 0x7f48441f06f0 2023-12-04T05:40:01.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.911+0000 7f48417fa640 1 -- 172.21.15.107:0/1921270470 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f482c0073c0 con 0x7f48441f06f0 2023-12-04T05:40:01.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.911+0000 7f48417fa640 1 --2- 172.21.15.107:0/1921270470 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f48240668d0 0x7f4824068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:01.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.911+0000 7f48417fa640 1 -- 172.21.15.107:0/1921270470 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f482c11ec10 con 0x7f48441f06f0 2023-12-04T05:40:01.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.911+0000 7f4843fff640 1 --2- 172.21.15.107:0/1921270470 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f48240668d0 0x7f4824068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:01.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.911+0000 7f4843fff640 1 --2- 172.21.15.107:0/1921270470 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f48240668d0 0x7f4824068d90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4834001230 tx=0x7f4834001160 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:01.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:01.915+0000 7f48417fa640 1 -- 172.21.15.107:0/1921270470 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f482c0ebe50 con 0x7f48441f06f0 2023-12-04T05:40:02.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.103+0000 7f484a73f640 1 -- 172.21.15.107:0/1921270470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f4844067ec0 con 0x7f48441f06f0 2023-12-04T05:40:02.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.751+0000 7f48417fa640 1 -- 172.21.15.107:0/1921270470 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f482c0c4020 con 0x7f48441f06f0 2023-12-04T05:40:02.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.755+0000 7f484a73f640 1 -- 172.21.15.107:0/1921270470 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f48240668d0 msgr2=0x7f4824068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:02.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.755+0000 7f484a73f640 1 --2- 172.21.15.107:0/1921270470 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f48240668d0 0x7f4824068d90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4834001230 tx=0x7f4834001160 comp rx=0 tx=0).stop 2023-12-04T05:40:02.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.755+0000 7f484a73f640 1 -- 172.21.15.107:0/1921270470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48441f06f0 msgr2=0x7f48440b2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:02.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.755+0000 7f484a73f640 1 --2- 172.21.15.107:0/1921270470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48441f06f0 0x7f48440b2e50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f482c005040 tx=0x7f4830005800 comp rx=0 tx=0).stop 2023-12-04T05:40:02.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.755+0000 7f484a73f640 1 -- 172.21.15.107:0/1921270470 shutdown_connections 2023-12-04T05:40:02.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.755+0000 7f484a73f640 1 --2- 172.21.15.107:0/1921270470 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f48240668d0 0x7f4824068d90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:02.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.755+0000 7f484a73f640 1 --2- 172.21.15.107:0/1921270470 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48441e5360 0x7f48440ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:02.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.755+0000 7f484a73f640 1 --2- 172.21.15.107:0/1921270470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48441f06f0 0x7f48440b2e50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:02.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.755+0000 7f484a73f640 1 --2- 172.21.15.107:0/1921270470 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48300047d0 0x7f48441e5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:02.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.755+0000 7f484a73f640 1 -- 172.21.15.107:0/1921270470 >> 172.21.15.107:0/1921270470 conn(0x7f484406b790 msgr2=0x7f484414ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:02.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.755+0000 7f484a73f640 1 -- 172.21.15.107:0/1921270470 shutdown_connections 2023-12-04T05:40:02.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.755+0000 7f484a73f640 1 -- 172.21.15.107:0/1921270470 wait complete. 2023-12-04T05:40:02.778 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-04T05:40:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.899+0000 7f15d37f2640 1 Processor -- start 2023-12-04T05:40:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.899+0000 7f15d37f2640 1 -- start start 2023-12-04T05:40:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15d37f2640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f15cc154020 0x7f15cc15a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15d37f2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f15cc0c4c10 0x7f15cc0c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15d37f2640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15cc15ad50 con 0x7f15cc0d01c0 2023-12-04T05:40:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15d37f2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15cc0ca230 con 0x7f15cc0c4c10 2023-12-04T05:40:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15d37f2640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15cc0c6770 con 0x7f15cc154020 2023-12-04T05:40:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15d1d8f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f15cc0c4c10 0x7f15cc0c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15d1d8f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f15cc0c4c10 0x7f15cc0c5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33802/0 (socket says 172.21.15.107:33802) 2023-12-04T05:40:02.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15d1d8f640 1 -- 172.21.15.107:0/1231720102 learned_addr learned my addr 172.21.15.107:0/1231720102 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:02.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15d1d8f640 1 -- 172.21.15.107:0/1231720102 >> v2:172.21.15.195:3300/0 conn(0x7f15cc154020 msgr2=0x7f15cc15a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:02.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15d158e640 1 --2- 172.21.15.107:0/1231720102 >> v2:172.21.15.195:3300/0 conn(0x7f15cc154020 0x7f15cc15a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:02.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15d1d8f640 1 --2- 172.21.15.107:0/1231720102 >> v2:172.21.15.195:3300/0 conn(0x7f15cc154020 0x7f15cc15a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:02.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15d1d8f640 1 -- 172.21.15.107:0/1231720102 >> v1:172.21.15.195:6789/0 conn(0x7f15cc0d01c0 legacy=0x7f15cc071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:02.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15d1d8f640 1 -- 172.21.15.107:0/1231720102 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15cc15d710 con 0x7f15cc0c4c10 2023-12-04T05:40:02.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15d158e640 1 --2- 172.21.15.107:0/1231720102 >> v2:172.21.15.195:3300/0 conn(0x7f15cc154020 0x7f15cc15a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:02.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15d1d8f640 1 --2- 172.21.15.107:0/1231720102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc0c4c10 0x7f15cc0c5010 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f15cc0d0150 tx=0x7f15c401f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a9c05c8e811e0d3 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:02.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15c3fff640 1 -- 172.21.15.107:0/1231720102 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c40af400 con 0x7f15cc0c4c10 2023-12-04T05:40:02.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15c3fff640 1 -- 172.21.15.107:0/1231720102 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c40ab050 con 0x7f15cc0c4c10 2023-12-04T05:40:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15c3fff640 1 -- 172.21.15.107:0/1231720102 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c40af780 con 0x7f15cc0c4c10 2023-12-04T05:40:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15d37f2640 1 -- 172.21.15.107:0/1231720102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc0c4c10 msgr2=0x7f15cc0c5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15d37f2640 1 --2- 172.21.15.107:0/1231720102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc0c4c10 0x7f15cc0c5010 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f15cc0d0150 tx=0x7f15c401f180 comp rx=0 tx=0).stop 2023-12-04T05:40:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15d37f2640 1 -- 172.21.15.107:0/1231720102 shutdown_connections 2023-12-04T05:40:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15d37f2640 1 --2- 172.21.15.107:0/1231720102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc0c4c10 0x7f15cc0c5010 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15d37f2640 1 --2- 172.21.15.107:0/1231720102 >> v2:172.21.15.195:3300/0 conn(0x7f15cc154020 0x7f15cc15a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.903+0000 7f15d37f2640 1 -- 172.21.15.107:0/1231720102 >> 172.21.15.107:0/1231720102 conn(0x7f15cc06c6c0 msgr2=0x7f15cc0cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d37f2640 1 -- 172.21.15.107:0/1231720102 shutdown_connections 2023-12-04T05:40:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d37f2640 1 -- 172.21.15.107:0/1231720102 wait complete. 2023-12-04T05:40:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d37f2640 1 Processor -- start 2023-12-04T05:40:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d37f2640 1 -- start start 2023-12-04T05:40:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d37f2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc0c4c10 0x7f15cc0c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d37f2640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15cc0d01c0 0x7f15cc0cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d37f2640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15cc154020 0x7f15cc0cb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d37f2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15cc0cac80 con 0x7f15cc0c4c10 2023-12-04T05:40:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d37f2640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15cc15d2b0 con 0x7f15cc154020 2023-12-04T05:40:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d37f2640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15cc0cb650 con 0x7f15cc0d01c0 2023-12-04T05:40:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d158e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc0c4c10 0x7f15cc0c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d158e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc0c4c10 0x7f15cc0c6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33818/0 (socket says 172.21.15.107:33818) 2023-12-04T05:40:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d158e640 1 -- 172.21.15.107:0/3607564051 learned_addr learned my addr 172.21.15.107:0/3607564051 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d1d8f640 1 --2- 172.21.15.107:0/3607564051 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15cc154020 0x7f15cc0cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d0d8d640 1 --2- 172.21.15.107:0/3607564051 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15cc0d01c0 0x7f15cc0cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d158e640 1 -- 172.21.15.107:0/3607564051 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15cc0d01c0 msgr2=0x7f15cc0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d158e640 1 --2- 172.21.15.107:0/3607564051 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15cc0d01c0 0x7f15cc0cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d158e640 1 -- 172.21.15.107:0/3607564051 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15cc154020 msgr2=0x7f15cc0cb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d158e640 1 --2- 172.21.15.107:0/3607564051 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15cc154020 0x7f15cc0cb110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d158e640 1 -- 172.21.15.107:0/3607564051 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c4009b30 con 0x7f15cc0c4c10 2023-12-04T05:40:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d1d8f640 1 --2- 172.21.15.107:0/3607564051 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15cc154020 0x7f15cc0cb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d0d8d640 1 --2- 172.21.15.107:0/3607564051 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15cc0d01c0 0x7f15cc0cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:02.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d158e640 1 --2- 172.21.15.107:0/3607564051 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc0c4c10 0x7f15cc0c6a90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f15c800c820 tx=0x7f15c800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:02.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15c1ffb640 1 -- 172.21.15.107:0/3607564051 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c8007a30 con 0x7f15cc0c4c10 2023-12-04T05:40:02.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15c1ffb640 1 -- 172.21.15.107:0/3607564051 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c8007b90 con 0x7f15cc0c4c10 2023-12-04T05:40:02.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d37f2640 1 -- 172.21.15.107:0/3607564051 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15cc15ca10 con 0x7f15cc0c4c10 2023-12-04T05:40:02.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15c1ffb640 1 -- 172.21.15.107:0/3607564051 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c800ce10 con 0x7f15cc0c4c10 2023-12-04T05:40:02.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d37f2640 1 -- 172.21.15.107:0/3607564051 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15cc15cfd0 con 0x7f15cc0c4c10 2023-12-04T05:40:02.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.907+0000 7f15d37f2640 1 -- 172.21.15.107:0/3607564051 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15cc067870 con 0x7f15cc0c4c10 2023-12-04T05:40:02.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.911+0000 7f15c1ffb640 1 -- 172.21.15.107:0/3607564051 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f15c80113f0 con 0x7f15cc0c4c10 2023-12-04T05:40:02.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.911+0000 7f15c1ffb640 1 --2- 172.21.15.107:0/3607564051 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f15b0066900 0x7f15b0068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:02.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.911+0000 7f15d0d8d640 1 --2- 172.21.15.107:0/3607564051 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f15b0066900 0x7f15b0068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:02.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.911+0000 7f15c1ffb640 1 -- 172.21.15.107:0/3607564051 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15c80845c0 con 0x7f15cc0c4c10 2023-12-04T05:40:02.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.911+0000 7f15d0d8d640 1 --2- 172.21.15.107:0/3607564051 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f15b0066900 0x7f15b0068dc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f15cc0cdb10 tx=0x7f15bc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:02.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:02.915+0000 7f15c1ffb640 1 -- 172.21.15.107:0/3607564051 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f15c8051770 con 0x7f15cc0c4c10 2023-12-04T05:40:03.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.103+0000 7f15d37f2640 1 -- 172.21.15.107:0/3607564051 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f15cc0c9bc0 con 0x7f15cc0c4c10 2023-12-04T05:40:03.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.767+0000 7f15c1ffb640 1 -- 172.21.15.107:0/3607564051 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7f15c8054d90 con 0x7f15cc0c4c10 2023-12-04T05:40:03.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.771+0000 7f15d37f2640 1 -- 172.21.15.107:0/3607564051 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f15b0066900 msgr2=0x7f15b0068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:03.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.771+0000 7f15d37f2640 1 --2- 172.21.15.107:0/3607564051 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f15b0066900 0x7f15b0068dc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f15cc0cdb10 tx=0x7f15bc009040 comp rx=0 tx=0).stop 2023-12-04T05:40:03.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.771+0000 7f15d37f2640 1 -- 172.21.15.107:0/3607564051 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc0c4c10 msgr2=0x7f15cc0c6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:03.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.771+0000 7f15d37f2640 1 --2- 172.21.15.107:0/3607564051 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc0c4c10 0x7f15cc0c6a90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f15c800c820 tx=0x7f15c800cce0 comp rx=0 tx=0).stop 2023-12-04T05:40:03.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.771+0000 7f15d37f2640 1 -- 172.21.15.107:0/3607564051 shutdown_connections 2023-12-04T05:40:03.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.771+0000 7f15d37f2640 1 --2- 172.21.15.107:0/3607564051 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f15b0066900 0x7f15b0068dc0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:03.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.771+0000 7f15d37f2640 1 --2- 172.21.15.107:0/3607564051 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15cc154020 0x7f15cc0cb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:03.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.771+0000 7f15d37f2640 1 --2- 172.21.15.107:0/3607564051 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15cc0d01c0 0x7f15cc0cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:03.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.771+0000 7f15d37f2640 1 --2- 172.21.15.107:0/3607564051 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc0c4c10 0x7f15cc0c6a90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:03.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.771+0000 7f15d37f2640 1 -- 172.21.15.107:0/3607564051 >> 172.21.15.107:0/3607564051 conn(0x7f15cc06c6c0 msgr2=0x7f15cc151800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:03.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.771+0000 7f15d37f2640 1 -- 172.21.15.107:0/3607564051 shutdown_connections 2023-12-04T05:40:03.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.771+0000 7f15d37f2640 1 -- 172.21.15.107:0/3607564051 wait complete. 2023-12-04T05:40:03.795 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-12-04T05:40:03.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb8321640 1 Processor -- start 2023-12-04T05:40:03.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb8321640 1 -- start start 2023-12-04T05:40:03.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb8321640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7fb0157d80 0x7f7fb00c5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:03.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb8321640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb00c3630 con 0x7f7fb0072680 2023-12-04T05:40:03.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb8321640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb00c4e00 con 0x7f7fb00b1b80 2023-12-04T05:40:03.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb8321640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb00c7180 con 0x7f7fb0157d80 2023-12-04T05:40:03.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb68be640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7fb0072680 0x7f7fb0162620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60304/0 (socket says 172.21.15.107:60304) 2023-12-04T05:40:03.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb68be640 1 -- 172.21.15.107:0/2996411761 learned_addr learned my addr 172.21.15.107:0/2996411761 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:03.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb58bc640 1 --2- 172.21.15.107:0/2996411761 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7fb0157d80 0x7f7fb00c5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:03.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb50bb640 1 -- 172.21.15.107:0/2996411761 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754032888 0 0) 0x7f7fb00c3630 con 0x7f7fb0072680 2023-12-04T05:40:03.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb50bb640 1 -- 172.21.15.107:0/2996411761 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f9c003530 con 0x7f7fb0072680 2023-12-04T05:40:03.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb50bb640 1 -- 172.21.15.107:0/2996411761 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 47911575 0 0) 0x7f7f9c003530 con 0x7f7fb0072680 2023-12-04T05:40:03.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb50bb640 1 -- 172.21.15.107:0/2996411761 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7fb0157d80 msgr2=0x7f7fb00c5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:03.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb50bb640 1 --2- 172.21.15.107:0/2996411761 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7fb0157d80 0x7f7fb00c5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:03.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb50bb640 1 -- 172.21.15.107:0/2996411761 >> v1:172.21.15.195:6789/0 conn(0x7f7fb00b1b80 legacy=0x7f7fb00c5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:03.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb50bb640 1 -- 172.21.15.107:0/2996411761 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb00c30b0 con 0x7f7fb0072680 2023-12-04T05:40:03.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb50bb640 1 -- 172.21.15.107:0/2996411761 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f7fa8001a00 con 0x7f7fb0072680 2023-12-04T05:40:03.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb50bb640 1 -- 172.21.15.107:0/2996411761 >> v1:172.21.15.107:6789/0 conn(0x7f7fb0072680 legacy=0x7f7fb0162620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:03.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb50bb640 1 --2- 172.21.15.107:0/2996411761 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7fb0157d80 0x7f7f9c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:03.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb50bb640 1 --2- 172.21.15.107:0/2996411761 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7f9c003f30 0x7f7f9c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:03.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb50bb640 1 --2- 172.21.15.107:0/2996411761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f9c004d10 0x7f7f9c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:03.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb50bb640 1 -- 172.21.15.107:0/2996411761 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb00c7180 con 0x7f7f9c004d10 2023-12-04T05:40:03.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb50bb640 1 -- 172.21.15.107:0/2996411761 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9c00ba70 con 0x7f7f9c003f30 2023-12-04T05:40:03.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb50bb640 1 -- 172.21.15.107:0/2996411761 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9c00bc30 con 0x7f7fb0157d80 2023-12-04T05:40:03.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb68be640 1 --2- 172.21.15.107:0/2996411761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f9c004d10 0x7f7f9c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb60bd640 1 --2- 172.21.15.107:0/2996411761 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7fb0157d80 0x7f7f9c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb58bc640 1 --2- 172.21.15.107:0/2996411761 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7f9c003f30 0x7f7f9c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb68be640 1 -- 172.21.15.107:0/2996411761 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7fb0157d80 msgr2=0x7f7f9c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb68be640 1 --2- 172.21.15.107:0/2996411761 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7fb0157d80 0x7f7f9c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb68be640 1 -- 172.21.15.107:0/2996411761 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7f9c003f30 msgr2=0x7f7f9c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb68be640 1 --2- 172.21.15.107:0/2996411761 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7f9c003f30 0x7f7f9c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.919+0000 7f7fb68be640 1 -- 172.21.15.107:0/2996411761 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9c003530 con 0x7f7f9c004d10 2023-12-04T05:40:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.923+0000 7f7fb68be640 1 --2- 172.21.15.107:0/2996411761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f9c004d10 0x7f7f9c005190 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7fa800a800 tx=0x7f7fa801d300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e66de1cb5651e42d server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.923+0000 7f7fb50bb640 1 -- 172.21.15.107:0/2996411761 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa80ad630 con 0x7f7f9c004d10 2023-12-04T05:40:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.923+0000 7f7fb50bb640 1 -- 172.21.15.107:0/2996411761 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa80a8070 con 0x7f7f9c004d10 2023-12-04T05:40:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.923+0000 7f7fb50bb640 1 -- 172.21.15.107:0/2996411761 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa80ada50 con 0x7f7f9c004d10 2023-12-04T05:40:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.923+0000 7f7fb8321640 1 -- 172.21.15.107:0/2996411761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f9c004d10 msgr2=0x7f7f9c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.923+0000 7f7fb8321640 1 --2- 172.21.15.107:0/2996411761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f9c004d10 0x7f7f9c005190 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7fa800a800 tx=0x7f7fa801d300 comp rx=0 tx=0).stop 2023-12-04T05:40:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.923+0000 7f7fb68be640 1 -- 172.21.15.107:0/2996411761 reap_dead start 2023-12-04T05:40:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.923+0000 7f7fb8321640 1 -- 172.21.15.107:0/2996411761 shutdown_connections 2023-12-04T05:40:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.923+0000 7f7fb8321640 1 -- 172.21.15.107:0/2996411761 >> 172.21.15.107:0/2996411761 conn(0x7f7fb006e710 msgr2=0x7f7fb00c8580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.923+0000 7f7fb8321640 1 -- 172.21.15.107:0/2996411761 shutdown_connections 2023-12-04T05:40:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.923+0000 7f7fb8321640 1 -- 172.21.15.107:0/2996411761 wait complete. 2023-12-04T05:40:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.923+0000 7f7fb8321640 1 Processor -- start 2023-12-04T05:40:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.923+0000 7f7fb8321640 1 -- start start 2023-12-04T05:40:03.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb8321640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7fb0151ef0 0x7f7fb0153c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:03.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb8321640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb014ede0 0x7f7fb014aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:03.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb8321640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7fb01522d0 0x7f7fb014b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:03.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb8321640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb0153870 con 0x7f7fb014ede0 2023-12-04T05:40:03.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb8321640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb01ed910 con 0x7f7fb01522d0 2023-12-04T05:40:03.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb8321640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb0152bf0 con 0x7f7fb0151ef0 2023-12-04T05:40:03.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb58bc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb014ede0 0x7f7fb014aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:03.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb58bc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb014ede0 0x7f7fb014aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33836/0 (socket says 172.21.15.107:33836) 2023-12-04T05:40:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb58bc640 1 -- 172.21.15.107:0/1214960282 learned_addr learned my addr 172.21.15.107:0/1214960282 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb60bd640 1 --2- 172.21.15.107:0/1214960282 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7fb0151ef0 0x7f7fb0153c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb68be640 1 --2- 172.21.15.107:0/1214960282 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7fb01522d0 0x7f7fb014b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb58bc640 1 -- 172.21.15.107:0/1214960282 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7fb0151ef0 msgr2=0x7f7fb0153c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb58bc640 1 --2- 172.21.15.107:0/1214960282 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7fb0151ef0 0x7f7fb0153c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb58bc640 1 -- 172.21.15.107:0/1214960282 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7fb01522d0 msgr2=0x7f7fb014b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb58bc640 1 --2- 172.21.15.107:0/1214960282 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7fb01522d0 0x7f7fb014b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb58bc640 1 -- 172.21.15.107:0/1214960282 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb0152ed0 con 0x7f7fb014ede0 2023-12-04T05:40:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb68be640 1 --2- 172.21.15.107:0/1214960282 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7fb01522d0 0x7f7fb014b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb58bc640 1 --2- 172.21.15.107:0/1214960282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb014ede0 0x7f7fb014aed0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7fa00115e0 tx=0x7f7fa0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fa6ffd640 1 -- 172.21.15.107:0/1214960282 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa000f8f0 con 0x7f7fb014ede0 2023-12-04T05:40:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fa6ffd640 1 -- 172.21.15.107:0/1214960282 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa000fa50 con 0x7f7fb014ede0 2023-12-04T05:40:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb8321640 1 -- 172.21.15.107:0/1214960282 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb01531b0 con 0x7f7fb014ede0 2023-12-04T05:40:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fa6ffd640 1 -- 172.21.15.107:0/1214960282 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa000fdc0 con 0x7f7fb014ede0 2023-12-04T05:40:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb8321640 1 -- 172.21.15.107:0/1214960282 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb01ed560 con 0x7f7fb014ede0 2023-12-04T05:40:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb8321640 1 -- 172.21.15.107:0/1214960282 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb01539b0 con 0x7f7fb014ede0 2023-12-04T05:40:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fa6ffd640 1 -- 172.21.15.107:0/1214960282 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f7fa0015460 con 0x7f7fb014ede0 2023-12-04T05:40:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fa6ffd640 1 --2- 172.21.15.107:0/1214960282 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7f94066900 0x7f7f94068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fa6ffd640 1 -- 172.21.15.107:0/1214960282 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7fa00849a0 con 0x7f7fb014ede0 2023-12-04T05:40:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb60bd640 1 --2- 172.21.15.107:0/1214960282 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7f94066900 0x7f7f94068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.927+0000 7f7fb60bd640 1 --2- 172.21.15.107:0/1214960282 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7f94066900 0x7f7f94068dc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7fac005510 tx=0x7f7fac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:03.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:03.931+0000 7f7fa6ffd640 1 -- 172.21.15.107:0/1214960282 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7fb01539b0 con 0x7f7fb014ede0 2023-12-04T05:40:04.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.107+0000 7f7fb8321640 1 -- 172.21.15.107:0/1214960282 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f7fb01539b0 con 0x7f7fb014ede0 2023-12-04T05:40:04.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.783+0000 7f7fa6ffd640 1 -- 172.21.15.107:0/1214960282 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v8) v1 ==== 115+0+0 (secure 0 0 0) 0x7f7fb01539b0 con 0x7f7fb014ede0 2023-12-04T05:40:04.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.787+0000 7f7fb8321640 1 -- 172.21.15.107:0/1214960282 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7f94066900 msgr2=0x7f7f94068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:04.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.787+0000 7f7fb8321640 1 --2- 172.21.15.107:0/1214960282 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7f94066900 0x7f7f94068dc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7fac005510 tx=0x7f7fac009040 comp rx=0 tx=0).stop 2023-12-04T05:40:04.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.787+0000 7f7fb8321640 1 -- 172.21.15.107:0/1214960282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb014ede0 msgr2=0x7f7fb014aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:04.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.787+0000 7f7fb8321640 1 --2- 172.21.15.107:0/1214960282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb014ede0 0x7f7fb014aed0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7fa00115e0 tx=0x7f7fa0011aa0 comp rx=0 tx=0).stop 2023-12-04T05:40:04.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.787+0000 7f7fb8321640 1 -- 172.21.15.107:0/1214960282 shutdown_connections 2023-12-04T05:40:04.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.787+0000 7f7fb8321640 1 --2- 172.21.15.107:0/1214960282 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7f94066900 0x7f7f94068dc0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:04.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.787+0000 7f7fb8321640 1 --2- 172.21.15.107:0/1214960282 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7fb01522d0 0x7f7fb014b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:04.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.787+0000 7f7fb8321640 1 --2- 172.21.15.107:0/1214960282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb014ede0 0x7f7fb014aed0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:04.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.787+0000 7f7fb8321640 1 --2- 172.21.15.107:0/1214960282 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7fb0151ef0 0x7f7fb0153c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:04.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.787+0000 7f7fb8321640 1 -- 172.21.15.107:0/1214960282 >> 172.21.15.107:0/1214960282 conn(0x7f7fb006e710 msgr2=0x7f7fb00c98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:04.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.787+0000 7f7fb8321640 1 -- 172.21.15.107:0/1214960282 shutdown_connections 2023-12-04T05:40:04.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.787+0000 7f7fb8321640 1 -- 172.21.15.107:0/1214960282 wait complete. 2023-12-04T05:40:04.810 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-12-04T05:40:04.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:40:04.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.931+0000 7f65c0737640 1 Processor -- start 2023-12-04T05:40:04.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.931+0000 7f65c0737640 1 -- start start 2023-12-04T05:40:04.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65c0737640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f65b8151180 0x7f65b80c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65c0737640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f65b80714f0 0x7f65b8071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65c0737640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b80700c0 con 0x7f65b8070380 2023-12-04T05:40:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65c0737640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b8071e60 con 0x7f65b80714f0 2023-12-04T05:40:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65c0737640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b8156df0 con 0x7f65b8151180 2023-12-04T05:40:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65bdcd2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f65b80714f0 0x7f65b8071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65bdcd2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f65b80714f0 0x7f65b8071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33852/0 (socket says 172.21.15.107:33852) 2023-12-04T05:40:04.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65bdcd2640 1 -- 172.21.15.107:0/3877479332 learned_addr learned my addr 172.21.15.107:0/3877479332 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:04.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65be4d3640 1 --2- 172.21.15.107:0/3877479332 >> v2:172.21.15.195:3300/0 conn(0x7f65b8151180 0x7f65b80c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:04.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65bdcd2640 1 -- 172.21.15.107:0/3877479332 >> v2:172.21.15.195:3300/0 conn(0x7f65b8151180 msgr2=0x7f65b80c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:04.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65bdcd2640 1 --2- 172.21.15.107:0/3877479332 >> v2:172.21.15.195:3300/0 conn(0x7f65b8151180 0x7f65b80c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:04.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65bdcd2640 1 -- 172.21.15.107:0/3877479332 >> v1:172.21.15.195:6789/0 conn(0x7f65b8070380 legacy=0x7f65b8070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:04.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65bdcd2640 1 -- 172.21.15.107:0/3877479332 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b81570d0 con 0x7f65b80714f0 2023-12-04T05:40:04.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65bdcd2640 1 --2- 172.21.15.107:0/3877479332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65b80714f0 0x7f65b8071920 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f65b806ba20 tx=0x7f65a801ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f091aa2e144e6ab server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:04.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65bd4d1640 1 -- 172.21.15.107:0/3877479332 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a80a89b0 con 0x7f65b80714f0 2023-12-04T05:40:04.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65bd4d1640 1 -- 172.21.15.107:0/3877479332 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a80a8b10 con 0x7f65b80714f0 2023-12-04T05:40:04.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65bd4d1640 1 -- 172.21.15.107:0/3877479332 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a80a8ea0 con 0x7f65b80714f0 2023-12-04T05:40:04.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65c0737640 1 -- 172.21.15.107:0/3877479332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65b80714f0 msgr2=0x7f65b8071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:04.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65c0737640 1 --2- 172.21.15.107:0/3877479332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65b80714f0 0x7f65b8071920 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f65b806ba20 tx=0x7f65a801ecb0 comp rx=0 tx=0).stop 2023-12-04T05:40:04.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65c0737640 1 -- 172.21.15.107:0/3877479332 shutdown_connections 2023-12-04T05:40:04.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65c0737640 1 --2- 172.21.15.107:0/3877479332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65b80714f0 0x7f65b8071920 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:04.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65c0737640 1 --2- 172.21.15.107:0/3877479332 >> v2:172.21.15.195:3300/0 conn(0x7f65b8151180 0x7f65b80c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:04.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.935+0000 7f65c0737640 1 -- 172.21.15.107:0/3877479332 >> 172.21.15.107:0/3877479332 conn(0x7f65b80c7ff0 msgr2=0x7f65b80c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:04.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65c0737640 1 -- 172.21.15.107:0/3877479332 shutdown_connections 2023-12-04T05:40:04.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65c0737640 1 -- 172.21.15.107:0/3877479332 wait complete. 2023-12-04T05:40:04.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65c0737640 1 Processor -- start 2023-12-04T05:40:04.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65c0737640 1 -- start start 2023-12-04T05:40:04.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65c0737640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65b8070380 0x7f65b8139e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:04.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65c0737640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f65b80714f0 0x7f65b815aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:04.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65c0737640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f65b8151180 0x7f65b81edc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:04.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65c0737640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b813df20 con 0x7f65b8070380 2023-12-04T05:40:04.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65c0737640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b81395e0 con 0x7f65b8151180 2023-12-04T05:40:04.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65c0737640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b81f19b0 con 0x7f65b80714f0 2023-12-04T05:40:04.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65be4d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65b8070380 0x7f65b8139e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:04.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65be4d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65b8070380 0x7f65b8139e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33854/0 (socket says 172.21.15.107:33854) 2023-12-04T05:40:04.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65be4d3640 1 -- 172.21.15.107:0/916568099 learned_addr learned my addr 172.21.15.107:0/916568099 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:04.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65becd4640 1 --2- 172.21.15.107:0/916568099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f65b8151180 0x7f65b81edc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:04.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65bdcd2640 1 --2- 172.21.15.107:0/916568099 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f65b80714f0 0x7f65b815aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:04.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65be4d3640 1 -- 172.21.15.107:0/916568099 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f65b80714f0 msgr2=0x7f65b815aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:04.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65be4d3640 1 --2- 172.21.15.107:0/916568099 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f65b80714f0 0x7f65b815aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:04.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65be4d3640 1 -- 172.21.15.107:0/916568099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f65b8151180 msgr2=0x7f65b81edc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:04.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65be4d3640 1 --2- 172.21.15.107:0/916568099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f65b8151180 0x7f65b81edc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:04.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65be4d3640 1 -- 172.21.15.107:0/916568099 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a80094d0 con 0x7f65b8070380 2023-12-04T05:40:04.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65bdcd2640 1 --2- 172.21.15.107:0/916568099 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f65b80714f0 0x7f65b815aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:04.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65be4d3640 1 --2- 172.21.15.107:0/916568099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65b8070380 0x7f65b8139e30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f65b400d5f0 tx=0x7f65b400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:04.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65aeffd640 1 -- 172.21.15.107:0/916568099 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b40125f0 con 0x7f65b8070380 2023-12-04T05:40:04.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65aeffd640 1 -- 172.21.15.107:0/916568099 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b4012750 con 0x7f65b8070380 2023-12-04T05:40:04.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65c0737640 1 -- 172.21.15.107:0/916568099 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b81f1c90 con 0x7f65b8070380 2023-12-04T05:40:04.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65aeffd640 1 -- 172.21.15.107:0/916568099 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4012ae0 con 0x7f65b8070380 2023-12-04T05:40:04.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65c0737640 1 -- 172.21.15.107:0/916568099 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b815b010 con 0x7f65b8070380 2023-12-04T05:40:04.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.939+0000 7f65c0737640 1 -- 172.21.15.107:0/916568099 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b81f1ed0 con 0x7f65b8070380 2023-12-04T05:40:04.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.943+0000 7f65aeffd640 1 -- 172.21.15.107:0/916568099 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f65b4012c40 con 0x7f65b8070380 2023-12-04T05:40:04.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.943+0000 7f65aeffd640 1 --2- 172.21.15.107:0/916568099 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f659c0668d0 0x7f659c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:04.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.943+0000 7f65aeffd640 1 -- 172.21.15.107:0/916568099 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65b40849d0 con 0x7f65b8070380 2023-12-04T05:40:04.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.943+0000 7f65bdcd2640 1 --2- 172.21.15.107:0/916568099 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f659c0668d0 0x7f659c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:04.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.943+0000 7f65bdcd2640 1 --2- 172.21.15.107:0/916568099 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f659c0668d0 0x7f659c068d90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f65a80b2000 tx=0x7f65a80aa070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:04.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:04.947+0000 7f65aeffd640 1 -- 172.21.15.107:0/916568099 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f65b4051b80 con 0x7f65b8070380 2023-12-04T05:40:05.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.131+0000 7f65c0737640 1 -- 172.21.15.107:0/916568099 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65b806d620 con 0x7f65b8070380 2023-12-04T05:40:05.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.135+0000 7f65aeffd640 1 -- 172.21.15.107:0/916568099 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6502 (secure 0 0 0) 0x7f65b40551a0 con 0x7f65b8070380 2023-12-04T05:40:05.136 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:05.136 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6836/3769196693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":3769196693},{"type":"v1","addr":"172.21.15.195:6836","nonce":3769196693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:40:04.110027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:40:05.137 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 8 2023-12-04T05:40:05.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.135+0000 7f65c0737640 1 -- 172.21.15.107:0/916568099 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f659c0668d0 msgr2=0x7f659c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:05.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.135+0000 7f65c0737640 1 --2- 172.21.15.107:0/916568099 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f659c0668d0 0x7f659c068d90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f65a80b2000 tx=0x7f65a80aa070 comp rx=0 tx=0).stop 2023-12-04T05:40:05.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.135+0000 7f65c0737640 1 -- 172.21.15.107:0/916568099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65b8070380 msgr2=0x7f65b8139e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:05.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.135+0000 7f65c0737640 1 --2- 172.21.15.107:0/916568099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65b8070380 0x7f65b8139e30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f65b400d5f0 tx=0x7f65b400dab0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.135+0000 7f65c0737640 1 -- 172.21.15.107:0/916568099 shutdown_connections 2023-12-04T05:40:05.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.135+0000 7f65c0737640 1 --2- 172.21.15.107:0/916568099 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f659c0668d0 0x7f659c068d90 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.135+0000 7f65c0737640 1 --2- 172.21.15.107:0/916568099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f65b8151180 0x7f65b81edc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.135+0000 7f65c0737640 1 --2- 172.21.15.107:0/916568099 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f65b80714f0 0x7f65b815aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.135+0000 7f65c0737640 1 --2- 172.21.15.107:0/916568099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65b8070380 0x7f65b8139e30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.135+0000 7f65c0737640 1 -- 172.21.15.107:0/916568099 >> 172.21.15.107:0/916568099 conn(0x7f65b80c7ff0 msgr2=0x7f65b814beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:05.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.135+0000 7f65c0737640 1 -- 172.21.15.107:0/916568099 shutdown_connections 2023-12-04T05:40:05.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.135+0000 7f65c0737640 1 -- 172.21.15.107:0/916568099 wait complete. 2023-12-04T05:40:05.155 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T05:40:05.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.275+0000 7f03b8800640 1 Processor -- start 2023-12-04T05:40:05.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.275+0000 7f03b8800640 1 -- start start 2023-12-04T05:40:05.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.279+0000 7f03b8800640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f03b00c46a0 0x7f03b00c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:05.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.279+0000 7f03b8800640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f03b0157180 0x7f03b00c7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:05.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.279+0000 7f03b8800640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b006e660 con 0x7f03b00c4a80 2023-12-04T05:40:05.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.279+0000 7f03b8800640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03b0156490 con 0x7f03b00c46a0 2023-12-04T05:40:05.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.279+0000 7f03b8800640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b0156cf0 con 0x7f03b0157180 2023-12-04T05:40:05.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.279+0000 7f03b659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f03b00c46a0 0x7f03b00c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:05.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.279+0000 7f03b659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f03b00c46a0 0x7f03b00c8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33864/0 (socket says 172.21.15.107:33864) 2023-12-04T05:40:05.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.279+0000 7f03b659c640 1 -- 172.21.15.107:0/2594052868 learned_addr learned my addr 172.21.15.107:0/2594052868 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:05.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.279+0000 7f03b659c640 1 -- 172.21.15.107:0/2594052868 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f03b0157180 msgr2=0x7f03b00c7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:05.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.279+0000 7f03b5d9b640 1 --2- 172.21.15.107:0/2594052868 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f03b0157180 0x7f03b00c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:05.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.279+0000 7f03b659c640 1 --2- 172.21.15.107:0/2594052868 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f03b0157180 0x7f03b00c7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.279+0000 7f03b659c640 1 -- 172.21.15.107:0/2594052868 >> v1:172.21.15.195:6789/0 conn(0x7f03b00c4a80 legacy=0x7f03b00c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:05.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.279+0000 7f03b659c640 1 -- 172.21.15.107:0/2594052868 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b006e0a0 con 0x7f03b00c46a0 2023-12-04T05:40:05.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.279+0000 7f03b5d9b640 1 --2- 172.21.15.107:0/2594052868 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f03b0157180 0x7f03b00c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:05.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.279+0000 7f03b659c640 1 --2- 172.21.15.107:0/2594052868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b00c46a0 0x7f03b00c8b10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009870 tx=0x7f03a401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee4dd2a2354b8f67 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:05.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.279+0000 7f03b559a640 1 -- 172.21.15.107:0/2594052868 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a40ac410 con 0x7f03b00c46a0 2023-12-04T05:40:05.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.279+0000 7f03b559a640 1 -- 172.21.15.107:0/2594052868 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a40ac570 con 0x7f03b00c46a0 2023-12-04T05:40:05.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.279+0000 7f03b559a640 1 -- 172.21.15.107:0/2594052868 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a40ac900 con 0x7f03b00c46a0 2023-12-04T05:40:05.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b8800640 1 -- 172.21.15.107:0/2594052868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b00c46a0 msgr2=0x7f03b00c8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:05.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b8800640 1 --2- 172.21.15.107:0/2594052868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b00c46a0 0x7f03b00c8b10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009870 tx=0x7f03a401c500 comp rx=0 tx=0).stop 2023-12-04T05:40:05.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b8800640 1 -- 172.21.15.107:0/2594052868 shutdown_connections 2023-12-04T05:40:05.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b8800640 1 --2- 172.21.15.107:0/2594052868 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f03b0157180 0x7f03b00c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b8800640 1 --2- 172.21.15.107:0/2594052868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b00c46a0 0x7f03b00c8b10 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b8800640 1 -- 172.21.15.107:0/2594052868 >> 172.21.15.107:0/2594052868 conn(0x7f03b00c7ff0 msgr2=0x7f03b00c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:05.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b8800640 1 -- 172.21.15.107:0/2594052868 shutdown_connections 2023-12-04T05:40:05.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b8800640 1 -- 172.21.15.107:0/2594052868 wait complete. 2023-12-04T05:40:05.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b8800640 1 Processor -- start 2023-12-04T05:40:05.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b8800640 1 -- start start 2023-12-04T05:40:05.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b8800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b00c46a0 0x7f03b0142090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:05.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b8800640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f03b00c4a80 0x7f03b0144f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:05.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b8800640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03b0157180 0x7f03b01454a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:05.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b8800640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b00c9e20 con 0x7f03b00c46a0 2023-12-04T05:40:05.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b8800640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b00cb350 con 0x7f03b0157180 2023-12-04T05:40:05.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b8800640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b01f6630 con 0x7f03b00c4a80 2023-12-04T05:40:05.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b00c46a0 0x7f03b0142090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:05.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b00c46a0 0x7f03b0142090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33870/0 (socket says 172.21.15.107:33870) 2023-12-04T05:40:05.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b659c640 1 -- 172.21.15.107:0/1842115968 learned_addr learned my addr 172.21.15.107:0/1842115968 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:05.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b5d9b640 1 --2- 172.21.15.107:0/1842115968 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f03b00c4a80 0x7f03b0144f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:05.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b6d9d640 1 --2- 172.21.15.107:0/1842115968 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03b0157180 0x7f03b01454a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:05.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b659c640 1 -- 172.21.15.107:0/1842115968 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f03b00c4a80 msgr2=0x7f03b0144f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:05.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b659c640 1 --2- 172.21.15.107:0/1842115968 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f03b00c4a80 0x7f03b0144f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b659c640 1 -- 172.21.15.107:0/1842115968 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03b0157180 msgr2=0x7f03b01454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:05.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b659c640 1 --2- 172.21.15.107:0/1842115968 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03b0157180 0x7f03b01454a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b659c640 1 -- 172.21.15.107:0/1842115968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a40094d0 con 0x7f03b00c46a0 2023-12-04T05:40:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b6d9d640 1 --2- 172.21.15.107:0/1842115968 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03b0157180 0x7f03b01454a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b5d9b640 1 --2- 172.21.15.107:0/1842115968 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f03b00c4a80 0x7f03b0144f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b659c640 1 --2- 172.21.15.107:0/1842115968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b00c46a0 0x7f03b0142090 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f03a40af040 tx=0x7f03a40192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03a2ffd640 1 -- 172.21.15.107:0/1842115968 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a40ac410 con 0x7f03b00c46a0 2023-12-04T05:40:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03a2ffd640 1 -- 172.21.15.107:0/1842115968 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a40b5650 con 0x7f03b00c46a0 2023-12-04T05:40:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b8800640 1 -- 172.21.15.107:0/1842115968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b01f5a20 con 0x7f03b00c46a0 2023-12-04T05:40:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03a2ffd640 1 -- 172.21.15.107:0/1842115968 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a40b5a00 con 0x7f03b00c46a0 2023-12-04T05:40:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.283+0000 7f03b8800640 1 -- 172.21.15.107:0/1842115968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b01f5fe0 con 0x7f03b00c46a0 2023-12-04T05:40:05.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.287+0000 7f03b8800640 1 -- 172.21.15.107:0/1842115968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b01f2310 con 0x7f03b00c46a0 2023-12-04T05:40:05.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.287+0000 7f03a2ffd640 1 -- 172.21.15.107:0/1842115968 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f03a40b3070 con 0x7f03b00c46a0 2023-12-04T05:40:05.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.287+0000 7f03a2ffd640 1 --2- 172.21.15.107:0/1842115968 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f03940668d0 0x7f0394068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:05.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.287+0000 7f03a2ffd640 1 -- 172.21.15.107:0/1842115968 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f03a411e220 con 0x7f03b00c46a0 2023-12-04T05:40:05.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.287+0000 7f03b5d9b640 1 --2- 172.21.15.107:0/1842115968 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f03940668d0 0x7f0394068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:05.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.287+0000 7f03b5d9b640 1 --2- 172.21.15.107:0/1842115968 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f03940668d0 0x7f0394068d90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f03b006b5e0 tx=0x7f03a8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:05.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.291+0000 7f03a2ffd640 1 -- 172.21.15.107:0/1842115968 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f03a40eb460 con 0x7f03b00c46a0 2023-12-04T05:40:05.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.443+0000 7f03b8800640 1 -- 172.21.15.107:0/1842115968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03b006e660 con 0x7f03b00c46a0 2023-12-04T05:40:05.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.447+0000 7f03a2ffd640 1 -- 172.21.15.107:0/1842115968 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f03b006e660 con 0x7f03b00c46a0 2023-12-04T05:40:05.448 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:05.448 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":21,"fsid":"603e856b-31ac-475a-8f35-21ab7faadd48","created":"2023-12-04T05:39:34.921690+0000","modified":"2023-12-04T05:40:00.671650+0000","last_up_change":"2023-12-04T05:39:46.235790+0000","last_in_change":"2023-12-04T05:39:37.587639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T05:39:47.567702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T05:39:57.644286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T05:39:58.029897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ab7111c-0ae8-4d41-b6db-e3743614322c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6817","nonce":768714279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6821","nonce":768714279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6829","nonce":768714279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6825","nonce":768714279}]},"public_addr":"172.21.15.107:6817/768714279","cluster_addr":"172.21.15.107:6821/768714279","heartbeat_back_addr":"172.21.15.107:6829/768714279","heartbeat_front_addr":"172.21.15.107:6825/768714279","state":["exists","up"]},{"osd":1,"uuid":"baef7a1f-8026-4fab-9e70-48465271034e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6801","nonce":2496355981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6803","nonce":2496355981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6807","nonce":2496355981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6805","nonce":2496355981}]},"public_addr":"172.21.15.107:6801/2496355981","cluster_addr":"172.21.15.107:6803/2496355981","heartbeat_back_addr":"172.21.15.107:6807/2496355981","heartbeat_front_addr":"172.21.15.107:6805/2496355981","state":["exists","up"]},{"osd":2,"uuid":"90e8cf24-7b54-484a-9d85-b44a9d3dfeab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6809","nonce":3097298224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6811","nonce":3097298224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6815","nonce":3097298224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6813","nonce":3097298224}]},"public_addr":"172.21.15.107:6809/3097298224","cluster_addr":"172.21.15.107:6811/3097298224","heartbeat_back_addr":"172.21.15.107:6815/3097298224","heartbeat_front_addr":"172.21.15.107:6813/3097298224","state":["exists","up"]},{"osd":3,"uuid":"54658731-2b9a-4d5e-ab13-7c1be7d34e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6820","nonce":3773076671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6824","nonce":3773076671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6831","nonce":3773076671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6828","nonce":3773076671}]},"public_addr":"172.21.15.107:6820/3773076671","cluster_addr":"172.21.15.107:6824/3773076671","heartbeat_back_addr":"172.21.15.107:6831/3773076671","heartbeat_front_addr":"172.21.15.107:6828/3773076671","state":["exists","up"]},{"osd":4,"uuid":"9a9dea13-8afe-485d-aa43-45eda691179b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6801","nonce":2747823542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6803","nonce":2747823542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6807","nonce":2747823542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6805","nonce":2747823542}]},"public_addr":"172.21.15.195:6801/2747823542","cluster_addr":"172.21.15.195:6803/2747823542","heartbeat_back_addr":"172.21.15.195:6807/2747823542","heartbeat_front_addr":"172.21.15.195:6805/2747823542","state":["exists","up"]},{"osd":5,"uuid":"db840d83-5a7d-468a-8aa2-1997945847c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6825","nonce":1484264540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6827","nonce":1484264540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6831","nonce":1484264540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6829","nonce":1484264540}]},"public_addr":"172.21.15.195:6825/1484264540","cluster_addr":"172.21.15.195:6827/1484264540","heartbeat_back_addr":"172.21.15.195:6831/1484264540","heartbeat_front_addr":"172.21.15.195:6829/1484264540","state":["exists","up"]},{"osd":6,"uuid":"38b5f165-bf91-467e-a615-4b4784337689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6811","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6813","nonce":2638471412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6817","nonce":2638471412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6823","nonce":2638471412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6821","nonce":2638471412}]},"public_addr":"172.21.15.195:6813/2638471412","cluster_addr":"172.21.15.195:6817/2638471412","heartbeat_back_addr":"172.21.15.195:6823/2638471412","heartbeat_front_addr":"172.21.15.195:6821/2638471412","state":["exists","up"]},{"osd":7,"uuid":"e5b3c596-494b-4df3-9437-bc2d52168ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6809","nonce":768189614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6812","nonce":768189614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6820","nonce":768189614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6816","nonce":768189614}]},"public_addr":"172.21.15.195:6809/768189614","cluster_addr":"172.21.15.195:6812/768189614","heartbeat_back_addr":"172.21.15.195:6820/768189614","heartbeat_front_addr":"172.21.15.195:6816/768189614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.209077+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.086601+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.221568+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.238524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.059477+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.148422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.079867+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.064916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T05:40:05.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.447+0000 7f03b8800640 1 -- 172.21.15.107:0/1842115968 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f03940668d0 msgr2=0x7f0394068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:05.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.447+0000 7f03b8800640 1 --2- 172.21.15.107:0/1842115968 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f03940668d0 0x7f0394068d90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f03b006b5e0 tx=0x7f03a8008040 comp rx=0 tx=0).stop 2023-12-04T05:40:05.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.447+0000 7f03b8800640 1 -- 172.21.15.107:0/1842115968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b00c46a0 msgr2=0x7f03b0142090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:05.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.447+0000 7f03b8800640 1 --2- 172.21.15.107:0/1842115968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b00c46a0 0x7f03b0142090 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f03a40af040 tx=0x7f03a40192a0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.447+0000 7f03b8800640 1 -- 172.21.15.107:0/1842115968 shutdown_connections 2023-12-04T05:40:05.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.447+0000 7f03b8800640 1 --2- 172.21.15.107:0/1842115968 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f03940668d0 0x7f0394068d90 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.447+0000 7f03b8800640 1 --2- 172.21.15.107:0/1842115968 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03b0157180 0x7f03b01454a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.447+0000 7f03b8800640 1 --2- 172.21.15.107:0/1842115968 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f03b00c4a80 0x7f03b0144f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.447+0000 7f03b8800640 1 --2- 172.21.15.107:0/1842115968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b00c46a0 0x7f03b0142090 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.447+0000 7f03b8800640 1 -- 172.21.15.107:0/1842115968 >> 172.21.15.107:0/1842115968 conn(0x7f03b00c7ff0 msgr2=0x7f03b0155020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:05.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.447+0000 7f03b8800640 1 -- 172.21.15.107:0/1842115968 shutdown_connections 2023-12-04T05:40:05.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.447+0000 7f03b8800640 1 -- 172.21.15.107:0/1842115968 wait complete. 2023-12-04T05:40:05.467 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-04T05:40:05.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-04T05:40:05.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.583+0000 7ffae9431640 1 Processor -- start 2023-12-04T05:40:05.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.583+0000 7ffae9431640 1 -- start start 2023-12-04T05:40:05.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.583+0000 7ffae9431640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ffae40ca2d0 0x7ffae4072570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:05.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.583+0000 7ffae9431640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae40c9570 con 0x7ffae40ca6b0 2023-12-04T05:40:05.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae9431640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae40c8c80 con 0x7ffae4072ab0 2023-12-04T05:40:05.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae9431640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae4073290 con 0x7ffae40ca2d0 2023-12-04T05:40:05.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffada7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ffae40ca6b0 0x7ffae40cd510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60316/0 (socket says 172.21.15.107:60316) 2023-12-04T05:40:05.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffada7fc640 1 -- 172.21.15.107:0/2857825751 learned_addr learned my addr 172.21.15.107:0/2857825751 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:05.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae2ffd640 1 --2- 172.21.15.107:0/2857825751 >> v2:172.21.15.195:3300/0 conn(0x7ffae40ca2d0 0x7ffae4072570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:05.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae27fc640 1 -- 172.21.15.107:0/2857825751 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974546848 0 0) 0x7ffae40c9570 con 0x7ffae40ca6b0 2023-12-04T05:40:05.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae27fc640 1 -- 172.21.15.107:0/2857825751 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad0003530 con 0x7ffae40ca6b0 2023-12-04T05:40:05.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae27fc640 1 -- 172.21.15.107:0/2857825751 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2703013619 0 0) 0x7ffad0003530 con 0x7ffae40ca6b0 2023-12-04T05:40:05.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae27fc640 1 -- 172.21.15.107:0/2857825751 >> v2:172.21.15.195:3300/0 conn(0x7ffae40ca2d0 msgr2=0x7ffae4072570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:05.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae27fc640 1 --2- 172.21.15.107:0/2857825751 >> v2:172.21.15.195:3300/0 conn(0x7ffae40ca2d0 0x7ffae4072570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae27fc640 1 -- 172.21.15.107:0/2857825751 >> v1:172.21.15.195:6789/0 conn(0x7ffae4072ab0 legacy=0x7ffae4151960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:05.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae27fc640 1 -- 172.21.15.107:0/2857825751 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae4072fb0 con 0x7ffae40ca6b0 2023-12-04T05:40:05.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae27fc640 1 -- 172.21.15.107:0/2857825751 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7ffacc001680 con 0x7ffae40ca6b0 2023-12-04T05:40:05.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae27fc640 1 -- 172.21.15.107:0/2857825751 >> v1:172.21.15.107:6789/0 conn(0x7ffae40ca6b0 legacy=0x7ffae40cd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:05.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae27fc640 1 --2- 172.21.15.107:0/2857825751 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffad0003980 0x7ffad0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:05.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae27fc640 1 --2- 172.21.15.107:0/2857825751 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffad00047d0 0x7ffad0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:05.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae27fc640 1 --2- 172.21.15.107:0/2857825751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffad0005190 0x7ffad000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:05.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae27fc640 1 -- 172.21.15.107:0/2857825751 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae4073290 con 0x7ffad0005190 2023-12-04T05:40:05.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae27fc640 1 -- 172.21.15.107:0/2857825751 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad000c040 con 0x7ffad0003980 2023-12-04T05:40:05.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae27fc640 1 -- 172.21.15.107:0/2857825751 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad000c200 con 0x7ffad00047d0 2023-12-04T05:40:05.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae37fe640 1 --2- 172.21.15.107:0/2857825751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffad0005190 0x7ffad000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:05.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae2ffd640 1 --2- 172.21.15.107:0/2857825751 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffad0003980 0x7ffad0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:05.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae37fe640 1 -- 172.21.15.107:0/2857825751 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffad00047d0 msgr2=0x7ffad0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:05.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae37fe640 1 --2- 172.21.15.107:0/2857825751 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffad00047d0 0x7ffad0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae37fe640 1 -- 172.21.15.107:0/2857825751 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffad0003980 msgr2=0x7ffad0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:05.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae37fe640 1 --2- 172.21.15.107:0/2857825751 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffad0003980 0x7ffad0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae37fe640 1 -- 172.21.15.107:0/2857825751 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad0003530 con 0x7ffad0005190 2023-12-04T05:40:05.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae37fe640 1 -- 172.21.15.107:0/2857825751 reap_dead start 2023-12-04T05:40:05.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae37fe640 1 --2- 172.21.15.107:0/2857825751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffad0005190 0x7ffad000b990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ffadc001e40 tx=0x7ffadc01c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd4c42785e124368 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:05.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae27fc640 1 -- 172.21.15.107:0/2857825751 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc003b80 con 0x7ffad0005190 2023-12-04T05:40:05.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae27fc640 1 -- 172.21.15.107:0/2857825751 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffadc003ce0 con 0x7ffad0005190 2023-12-04T05:40:05.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.587+0000 7ffae27fc640 1 -- 172.21.15.107:0/2857825751 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc005260 con 0x7ffad0005190 2023-12-04T05:40:05.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae9431640 1 -- 172.21.15.107:0/2857825751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffad0005190 msgr2=0x7ffad000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:05.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae9431640 1 --2- 172.21.15.107:0/2857825751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffad0005190 0x7ffad000b990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ffadc001e40 tx=0x7ffadc01c740 comp rx=0 tx=0).stop 2023-12-04T05:40:05.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae9431640 1 -- 172.21.15.107:0/2857825751 shutdown_connections 2023-12-04T05:40:05.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae9431640 1 --2- 172.21.15.107:0/2857825751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffad0005190 0x7ffad000b990 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae9431640 1 -- 172.21.15.107:0/2857825751 >> 172.21.15.107:0/2857825751 conn(0x7ffae40663a0 msgr2=0x7ffae40667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:05.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae9431640 1 -- 172.21.15.107:0/2857825751 shutdown_connections 2023-12-04T05:40:05.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae9431640 1 -- 172.21.15.107:0/2857825751 wait complete. 2023-12-04T05:40:05.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae9431640 1 Processor -- start 2023-12-04T05:40:05.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae9431640 1 -- start start 2023-12-04T05:40:05.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae9431640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffad0005190 0x7ffae4156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:05.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae9431640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffae414b6a0 0x7ffae414c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:05.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae9431640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffae41f2e90 0x7ffae414c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:05.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae9431640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae406c970 con 0x7ffae41f2e90 2023-12-04T05:40:05.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae9431640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae41efc50 con 0x7ffad0005190 2023-12-04T05:40:05.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae9431640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae41eff30 con 0x7ffae414b6a0 2023-12-04T05:40:05.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffae41f2e90 0x7ffae414c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:05.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffae41f2e90 0x7ffae414c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33892/0 (socket says 172.21.15.107:33892) 2023-12-04T05:40:05.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae37fe640 1 -- 172.21.15.107:0/3168595225 learned_addr learned my addr 172.21.15.107:0/3168595225 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:05.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffada7fc640 1 --2- 172.21.15.107:0/3168595225 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffae414b6a0 0x7ffae414c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:05.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae2ffd640 1 --2- 172.21.15.107:0/3168595225 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffad0005190 0x7ffae4156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:05.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae37fe640 1 -- 172.21.15.107:0/3168595225 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffae414b6a0 msgr2=0x7ffae414c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:05.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae37fe640 1 --2- 172.21.15.107:0/3168595225 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffae414b6a0 0x7ffae414c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae37fe640 1 -- 172.21.15.107:0/3168595225 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffad0005190 msgr2=0x7ffae4156fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:05.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae37fe640 1 --2- 172.21.15.107:0/3168595225 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffad0005190 0x7ffae4156fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae37fe640 1 -- 172.21.15.107:0/3168595225 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae41f0210 con 0x7ffae41f2e90 2023-12-04T05:40:05.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae37fe640 1 --2- 172.21.15.107:0/3168595225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffae41f2e90 0x7ffae414c900 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ffadc01c570 tx=0x7ffad0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:05.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffadbfff640 1 -- 172.21.15.107:0/3168595225 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc004fb0 con 0x7ffae41f2e90 2023-12-04T05:40:05.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffadbfff640 1 -- 172.21.15.107:0/3168595225 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffadc005110 con 0x7ffae41f2e90 2023-12-04T05:40:05.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae9431640 1 -- 172.21.15.107:0/3168595225 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae406c970 con 0x7ffae41f2e90 2023-12-04T05:40:05.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffadbfff640 1 -- 172.21.15.107:0/3168595225 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc0054a0 con 0x7ffae41f2e90 2023-12-04T05:40:05.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae9431640 1 -- 172.21.15.107:0/3168595225 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae41ef920 con 0x7ffae41f2e90 2023-12-04T05:40:05.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.591+0000 7ffae9431640 1 -- 172.21.15.107:0/3168595225 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae414c0e0 con 0x7ffae41f2e90 2023-12-04T05:40:05.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.595+0000 7ffadbfff640 1 -- 172.21.15.107:0/3168595225 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ffadc005600 con 0x7ffae41f2e90 2023-12-04T05:40:05.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.595+0000 7ffadbfff640 1 --2- 172.21.15.107:0/3168595225 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ffac40668d0 0x7ffac4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:05.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.595+0000 7ffadbfff640 1 -- 172.21.15.107:0/3168595225 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffadc11d4e0 con 0x7ffae41f2e90 2023-12-04T05:40:05.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.595+0000 7ffae2ffd640 1 --2- 172.21.15.107:0/3168595225 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ffac40668d0 0x7ffac4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:05.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.595+0000 7ffae2ffd640 1 --2- 172.21.15.107:0/3168595225 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ffac40668d0 0x7ffac4068d90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffad40099b0 tx=0x7ffad4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:05.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.599+0000 7ffadbfff640 1 -- 172.21.15.107:0/3168595225 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ffae414c0e0 con 0x7ffae41f2e90 2023-12-04T05:40:05.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.739+0000 7ffae9431640 1 -- 172.21.15.107:0/3168595225 --> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffae40c9200 con 0x7ffac40668d0 2023-12-04T05:40:05.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.751+0000 7ffadbfff640 1 -- 172.21.15.107:0/3168595225 <== mgr.4104 v2:172.21.15.107:6832/2693048026 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264484 (secure 0 0 0) 0x7ffad4013960 con 0x7ffac40668d0 2023-12-04T05:40:05.757 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:05.758 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-12-04T05:40:05.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.755+0000 7ffae9431640 1 -- 172.21.15.107:0/3168595225 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ffac40668d0 msgr2=0x7ffac4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:05.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.755+0000 7ffae9431640 1 --2- 172.21.15.107:0/3168595225 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ffac40668d0 0x7ffac4068d90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffad40099b0 tx=0x7ffad4008040 comp rx=0 tx=0).stop 2023-12-04T05:40:05.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.755+0000 7ffae9431640 1 -- 172.21.15.107:0/3168595225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffae41f2e90 msgr2=0x7ffae414c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:05.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.755+0000 7ffae9431640 1 --2- 172.21.15.107:0/3168595225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffae41f2e90 0x7ffae414c900 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ffadc01c570 tx=0x7ffad0005800 comp rx=0 tx=0).stop 2023-12-04T05:40:05.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.755+0000 7ffae9431640 1 -- 172.21.15.107:0/3168595225 shutdown_connections 2023-12-04T05:40:05.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.755+0000 7ffae9431640 1 --2- 172.21.15.107:0/3168595225 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ffac40668d0 0x7ffac4068d90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.755+0000 7ffae9431640 1 --2- 172.21.15.107:0/3168595225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffae41f2e90 0x7ffae414c900 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.755+0000 7ffae9431640 1 --2- 172.21.15.107:0/3168595225 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffae414b6a0 0x7ffae414c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.755+0000 7ffae9431640 1 --2- 172.21.15.107:0/3168595225 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffad0005190 0x7ffae4156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.755+0000 7ffae9431640 1 -- 172.21.15.107:0/3168595225 >> 172.21.15.107:0/3168595225 conn(0x7ffae40663a0 msgr2=0x7ffae40cd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:05.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.755+0000 7ffae9431640 1 -- 172.21.15.107:0/3168595225 shutdown_connections 2023-12-04T05:40:05.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.755+0000 7ffae9431640 1 -- 172.21.15.107:0/3168595225 wait complete. 2023-12-04T05:40:05.774 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-04T05:40:05.492635+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":86,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235636,"kb_used_data":2808,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739084,"statfs":{"total":773094113280,"available":772852822016,"internally_reserved":0,"allocated":2875392,"data_stored":1760184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":67,"apply_latency_ms":67,"commit_latency_ns":67000000,"apply_latency_ns":67000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.799872"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503577+0000","last_change":"2023-12-04T05:39:59.736570+0000","last_active":"2023-12-04T05:40:01.503577+0000","last_peered":"2023-12-04T05:40:01.503577+0000","last_clean":"2023-12-04T05:40:01.503577+0000","last_became_active":"2023-12-04T05:39:59.736000+0000","last_became_peered":"2023-12-04T05:39:59.736000+0000","last_unstale":"2023-12-04T05:40:01.503577+0000","last_undegraded":"2023-12-04T05:40:01.503577+0000","last_fullsized":"2023-12-04T05:40:01.503577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:34:15.458889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908781+0000","last_change":"2023-12-04T05:39:59.703955+0000","last_active":"2023-12-04T05:40:00.908781+0000","last_peered":"2023-12-04T05:40:00.908781+0000","last_clean":"2023-12-04T05:40:00.908781+0000","last_became_active":"2023-12-04T05:39:59.703032+0000","last_became_peered":"2023-12-04T05:39:59.703032+0000","last_unstale":"2023-12-04T05:40:00.908781+0000","last_undegraded":"2023-12-04T05:40:00.908781+0000","last_fullsized":"2023-12-04T05:40:00.908781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:22:09.010348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501353+0000","last_change":"2023-12-04T05:39:59.705340+0000","last_active":"2023-12-04T05:40:01.501353+0000","last_peered":"2023-12-04T05:40:01.501353+0000","last_clean":"2023-12-04T05:40:01.501353+0000","last_became_active":"2023-12-04T05:39:59.699479+0000","last_became_peered":"2023-12-04T05:39:59.699479+0000","last_unstale":"2023-12-04T05:40:01.501353+0000","last_undegraded":"2023-12-04T05:40:01.501353+0000","last_fullsized":"2023-12-04T05:40:01.501353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:12:00.122168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683148+0000","last_change":"2023-12-04T05:39:59.728698+0000","last_active":"2023-12-04T05:40:00.683148+0000","last_peered":"2023-12-04T05:40:00.683148+0000","last_clean":"2023-12-04T05:40:00.683148+0000","last_became_active":"2023-12-04T05:39:59.727914+0000","last_became_peered":"2023-12-04T05:39:59.727914+0000","last_unstale":"2023-12-04T05:40:00.683148+0000","last_undegraded":"2023-12-04T05:40:00.683148+0000","last_fullsized":"2023-12-04T05:40:00.683148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:39:17.269322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502651+0000","last_change":"2023-12-04T05:39:59.726320+0000","last_active":"2023-12-04T05:40:01.502651+0000","last_peered":"2023-12-04T05:40:01.502651+0000","last_clean":"2023-12-04T05:40:01.502651+0000","last_became_active":"2023-12-04T05:39:59.722627+0000","last_became_peered":"2023-12-04T05:39:59.722627+0000","last_unstale":"2023-12-04T05:40:01.502651+0000","last_undegraded":"2023-12-04T05:40:01.502651+0000","last_fullsized":"2023-12-04T05:40:01.502651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:40:48.940938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682860+0000","last_change":"2023-12-04T05:39:59.718363+0000","last_active":"2023-12-04T05:40:00.682860+0000","last_peered":"2023-12-04T05:40:00.682860+0000","last_clean":"2023-12-04T05:40:00.682860+0000","last_became_active":"2023-12-04T05:39:59.717475+0000","last_became_peered":"2023-12-04T05:39:59.717475+0000","last_unstale":"2023-12-04T05:40:00.682860+0000","last_undegraded":"2023-12-04T05:40:00.682860+0000","last_fullsized":"2023-12-04T05:40:00.682860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:02:56.684095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501353+0000","last_change":"2023-12-04T05:39:59.724035+0000","last_active":"2023-12-04T05:40:01.501353+0000","last_peered":"2023-12-04T05:40:01.501353+0000","last_clean":"2023-12-04T05:40:01.501353+0000","last_became_active":"2023-12-04T05:39:59.723277+0000","last_became_peered":"2023-12-04T05:39:59.723277+0000","last_unstale":"2023-12-04T05:40:01.501353+0000","last_undegraded":"2023-12-04T05:40:01.501353+0000","last_fullsized":"2023-12-04T05:40:01.501353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:34:54.379064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502157+0000","last_change":"2023-12-04T05:39:59.720028+0000","last_active":"2023-12-04T05:40:01.502157+0000","last_peered":"2023-12-04T05:40:01.502157+0000","last_clean":"2023-12-04T05:40:01.502157+0000","last_became_active":"2023-12-04T05:39:59.719236+0000","last_became_peered":"2023-12-04T05:39:59.719236+0000","last_unstale":"2023-12-04T05:40:01.502157+0000","last_undegraded":"2023-12-04T05:40:01.502157+0000","last_fullsized":"2023-12-04T05:40:01.502157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:23:39.909425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503754+0000","last_change":"2023-12-04T05:39:59.732301+0000","last_active":"2023-12-04T05:40:01.503754+0000","last_peered":"2023-12-04T05:40:01.503754+0000","last_clean":"2023-12-04T05:40:01.503754+0000","last_became_active":"2023-12-04T05:39:59.731561+0000","last_became_peered":"2023-12-04T05:39:59.731561+0000","last_unstale":"2023-12-04T05:40:01.503754+0000","last_undegraded":"2023-12-04T05:40:01.503754+0000","last_fullsized":"2023-12-04T05:40:01.503754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:49:40.179867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501688+0000","last_change":"2023-12-04T05:39:59.721633+0000","last_active":"2023-12-04T05:40:01.501688+0000","last_peered":"2023-12-04T05:40:01.501688+0000","last_clean":"2023-12-04T05:40:01.501688+0000","last_became_active":"2023-12-04T05:39:59.720846+0000","last_became_peered":"2023-12-04T05:39:59.720846+0000","last_unstale":"2023-12-04T05:40:01.501688+0000","last_undegraded":"2023-12-04T05:40:01.501688+0000","last_fullsized":"2023-12-04T05:40:01.501688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:44:34.893307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501835+0000","last_change":"2023-12-04T05:39:59.721173+0000","last_active":"2023-12-04T05:40:01.501835+0000","last_peered":"2023-12-04T05:40:01.501835+0000","last_clean":"2023-12-04T05:40:01.501835+0000","last_became_active":"2023-12-04T05:39:59.720371+0000","last_became_peered":"2023-12-04T05:39:59.720371+0000","last_unstale":"2023-12-04T05:40:01.501835+0000","last_undegraded":"2023-12-04T05:40:01.501835+0000","last_fullsized":"2023-12-04T05:40:01.501835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:12:23.683644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682272+0000","last_change":"2023-12-04T05:39:59.710269+0000","last_active":"2023-12-04T05:40:00.682272+0000","last_peered":"2023-12-04T05:40:00.682272+0000","last_clean":"2023-12-04T05:40:00.682272+0000","last_became_active":"2023-12-04T05:39:59.709565+0000","last_became_peered":"2023-12-04T05:39:59.709565+0000","last_unstale":"2023-12-04T05:40:00.682272+0000","last_undegraded":"2023-12-04T05:40:00.682272+0000","last_fullsized":"2023-12-04T05:40:00.682272+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:11:13.222803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501884+0000","last_change":"2023-12-04T05:39:59.710484+0000","last_active":"2023-12-04T05:40:01.501884+0000","last_peered":"2023-12-04T05:40:01.501884+0000","last_clean":"2023-12-04T05:40:01.501884+0000","last_became_active":"2023-12-04T05:39:59.709623+0000","last_became_peered":"2023-12-04T05:39:59.709623+0000","last_unstale":"2023-12-04T05:40:01.501884+0000","last_undegraded":"2023-12-04T05:40:01.501884+0000","last_fullsized":"2023-12-04T05:40:01.501884+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:40:05.852673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.500880+0000","last_change":"2023-12-04T05:39:59.715685+0000","last_active":"2023-12-04T05:40:01.500880+0000","last_peered":"2023-12-04T05:40:01.500880+0000","last_clean":"2023-12-04T05:40:01.500880+0000","last_became_active":"2023-12-04T05:39:59.714890+0000","last_became_peered":"2023-12-04T05:39:59.714890+0000","last_unstale":"2023-12-04T05:40:01.500880+0000","last_undegraded":"2023-12-04T05:40:01.500880+0000","last_fullsized":"2023-12-04T05:40:01.500880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:59:34.243902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908703+0000","last_change":"2023-12-04T05:39:59.688405+0000","last_active":"2023-12-04T05:40:00.908703+0000","last_peered":"2023-12-04T05:40:00.908703+0000","last_clean":"2023-12-04T05:40:00.908703+0000","last_became_active":"2023-12-04T05:39:59.687603+0000","last_became_peered":"2023-12-04T05:39:59.687603+0000","last_unstale":"2023-12-04T05:40:00.908703+0000","last_undegraded":"2023-12-04T05:40:00.908703+0000","last_fullsized":"2023-12-04T05:40:00.908703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:44:51.269794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681203+0000","last_change":"2023-12-04T05:39:59.718543+0000","last_active":"2023-12-04T05:40:00.681203+0000","last_peered":"2023-12-04T05:40:00.681203+0000","last_clean":"2023-12-04T05:40:00.681203+0000","last_became_active":"2023-12-04T05:39:59.717826+0000","last_became_peered":"2023-12-04T05:39:59.717826+0000","last_unstale":"2023-12-04T05:40:00.681203+0000","last_undegraded":"2023-12-04T05:40:00.681203+0000","last_fullsized":"2023-12-04T05:40:00.681203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:31:03.248114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.684087+0000","last_change":"2023-12-04T05:39:59.724278+0000","last_active":"2023-12-04T05:40:00.684087+0000","last_peered":"2023-12-04T05:40:00.684087+0000","last_clean":"2023-12-04T05:40:00.684087+0000","last_became_active":"2023-12-04T05:39:59.723384+0000","last_became_peered":"2023-12-04T05:39:59.723384+0000","last_unstale":"2023-12-04T05:40:00.684087+0000","last_undegraded":"2023-12-04T05:40:00.684087+0000","last_fullsized":"2023-12-04T05:40:00.684087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:07:20.576260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.679022+0000","last_change":"2023-12-04T05:39:59.711802+0000","last_active":"2023-12-04T05:40:00.679022+0000","last_peered":"2023-12-04T05:40:00.679022+0000","last_clean":"2023-12-04T05:40:00.679022+0000","last_became_active":"2023-12-04T05:39:59.711044+0000","last_became_peered":"2023-12-04T05:39:59.711044+0000","last_unstale":"2023-12-04T05:40:00.679022+0000","last_undegraded":"2023-12-04T05:40:00.679022+0000","last_fullsized":"2023-12-04T05:40:00.679022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:42:28.221010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503300+0000","last_change":"2023-12-04T05:39:59.725226+0000","last_active":"2023-12-04T05:40:01.503300+0000","last_peered":"2023-12-04T05:40:01.503300+0000","last_clean":"2023-12-04T05:40:01.503300+0000","last_became_active":"2023-12-04T05:39:59.724334+0000","last_became_peered":"2023-12-04T05:39:59.724334+0000","last_unstale":"2023-12-04T05:40:01.503300+0000","last_undegraded":"2023-12-04T05:40:01.503300+0000","last_fullsized":"2023-12-04T05:40:01.503300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:48:30.439285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501624+0000","last_change":"2023-12-04T05:39:59.729858+0000","last_active":"2023-12-04T05:40:01.501624+0000","last_peered":"2023-12-04T05:40:01.501624+0000","last_clean":"2023-12-04T05:40:01.501624+0000","last_became_active":"2023-12-04T05:39:59.728989+0000","last_became_peered":"2023-12-04T05:39:59.728989+0000","last_unstale":"2023-12-04T05:40:01.501624+0000","last_undegraded":"2023-12-04T05:40:01.501624+0000","last_fullsized":"2023-12-04T05:40:01.501624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:42:01.099743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501616+0000","last_change":"2023-12-04T05:39:59.724022+0000","last_active":"2023-12-04T05:40:01.501616+0000","last_peered":"2023-12-04T05:40:01.501616+0000","last_clean":"2023-12-04T05:40:01.501616+0000","last_became_active":"2023-12-04T05:39:59.723120+0000","last_became_peered":"2023-12-04T05:39:59.723120+0000","last_unstale":"2023-12-04T05:40:01.501616+0000","last_undegraded":"2023-12-04T05:40:01.501616+0000","last_fullsized":"2023-12-04T05:40:01.501616+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:13:50.048757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683322+0000","last_change":"2023-12-04T05:39:59.714806+0000","last_active":"2023-12-04T05:40:00.683322+0000","last_peered":"2023-12-04T05:40:00.683322+0000","last_clean":"2023-12-04T05:40:00.683322+0000","last_became_active":"2023-12-04T05:39:59.713917+0000","last_became_peered":"2023-12-04T05:39:59.713917+0000","last_unstale":"2023-12-04T05:40:00.683322+0000","last_undegraded":"2023-12-04T05:40:00.683322+0000","last_fullsized":"2023-12-04T05:40:00.683322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:18:03.128562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501966+0000","last_change":"2023-12-04T05:39:59.729098+0000","last_active":"2023-12-04T05:40:01.501966+0000","last_peered":"2023-12-04T05:40:01.501966+0000","last_clean":"2023-12-04T05:40:01.501966+0000","last_became_active":"2023-12-04T05:39:59.728449+0000","last_became_peered":"2023-12-04T05:39:59.728449+0000","last_unstale":"2023-12-04T05:40:01.501966+0000","last_undegraded":"2023-12-04T05:40:01.501966+0000","last_fullsized":"2023-12-04T05:40:01.501966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:25:12.098431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682551+0000","last_change":"2023-12-04T05:39:59.708300+0000","last_active":"2023-12-04T05:40:00.682551+0000","last_peered":"2023-12-04T05:40:00.682551+0000","last_clean":"2023-12-04T05:40:00.682551+0000","last_became_active":"2023-12-04T05:39:59.707530+0000","last_became_peered":"2023-12-04T05:39:59.707530+0000","last_unstale":"2023-12-04T05:40:00.682551+0000","last_undegraded":"2023-12-04T05:40:00.682551+0000","last_fullsized":"2023-12-04T05:40:00.682551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:11:40.053312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502562+0000","last_change":"2023-12-04T05:39:59.729406+0000","last_active":"2023-12-04T05:40:01.502562+0000","last_peered":"2023-12-04T05:40:01.502562+0000","last_clean":"2023-12-04T05:40:01.502562+0000","last_became_active":"2023-12-04T05:39:59.728677+0000","last_became_peered":"2023-12-04T05:39:59.728677+0000","last_unstale":"2023-12-04T05:40:01.502562+0000","last_undegraded":"2023-12-04T05:40:01.502562+0000","last_fullsized":"2023-12-04T05:40:01.502562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:03:26.954387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501922+0000","last_change":"2023-12-04T05:39:59.733628+0000","last_active":"2023-12-04T05:40:01.501922+0000","last_peered":"2023-12-04T05:40:01.501922+0000","last_clean":"2023-12-04T05:40:01.501922+0000","last_became_active":"2023-12-04T05:39:59.726137+0000","last_became_peered":"2023-12-04T05:39:59.726137+0000","last_unstale":"2023-12-04T05:40:01.501922+0000","last_undegraded":"2023-12-04T05:40:01.501922+0000","last_fullsized":"2023-12-04T05:40:01.501922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:49:17.630085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908788+0000","last_change":"2023-12-04T05:39:59.727169+0000","last_active":"2023-12-04T05:40:00.908788+0000","last_peered":"2023-12-04T05:40:00.908788+0000","last_clean":"2023-12-04T05:40:00.908788+0000","last_became_active":"2023-12-04T05:39:59.725378+0000","last_became_peered":"2023-12-04T05:39:59.725378+0000","last_unstale":"2023-12-04T05:40:00.908788+0000","last_undegraded":"2023-12-04T05:40:00.908788+0000","last_fullsized":"2023-12-04T05:40:00.908788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:23:21.007397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502427+0000","last_change":"2023-12-04T05:39:59.729516+0000","last_active":"2023-12-04T05:40:01.502427+0000","last_peered":"2023-12-04T05:40:01.502427+0000","last_clean":"2023-12-04T05:40:01.502427+0000","last_became_active":"2023-12-04T05:39:59.728835+0000","last_became_peered":"2023-12-04T05:39:59.728835+0000","last_unstale":"2023-12-04T05:40:01.502427+0000","last_undegraded":"2023-12-04T05:40:01.502427+0000","last_fullsized":"2023-12-04T05:40:01.502427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:37:37.490362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501933+0000","last_change":"2023-12-04T05:39:59.724592+0000","last_active":"2023-12-04T05:40:01.501933+0000","last_peered":"2023-12-04T05:40:01.501933+0000","last_clean":"2023-12-04T05:40:01.501933+0000","last_became_active":"2023-12-04T05:39:59.723876+0000","last_became_peered":"2023-12-04T05:39:59.723876+0000","last_unstale":"2023-12-04T05:40:01.501933+0000","last_undegraded":"2023-12-04T05:40:01.501933+0000","last_fullsized":"2023-12-04T05:40:01.501933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:09:26.625728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683703+0000","last_change":"2023-12-04T05:39:59.726540+0000","last_active":"2023-12-04T05:40:00.683703+0000","last_peered":"2023-12-04T05:40:00.683703+0000","last_clean":"2023-12-04T05:40:00.683703+0000","last_became_active":"2023-12-04T05:39:59.725781+0000","last_became_peered":"2023-12-04T05:39:59.725781+0000","last_unstale":"2023-12-04T05:40:00.683703+0000","last_undegraded":"2023-12-04T05:40:00.683703+0000","last_fullsized":"2023-12-04T05:40:00.683703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:59:22.013410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.678751+0000","last_change":"2023-12-04T05:39:59.708376+0000","last_active":"2023-12-04T05:40:00.678751+0000","last_peered":"2023-12-04T05:40:00.678751+0000","last_clean":"2023-12-04T05:40:00.678751+0000","last_became_active":"2023-12-04T05:39:59.707604+0000","last_became_peered":"2023-12-04T05:39:59.707604+0000","last_unstale":"2023-12-04T05:40:00.678751+0000","last_undegraded":"2023-12-04T05:40:00.678751+0000","last_fullsized":"2023-12-04T05:40:00.678751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:00:23.385619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501476+0000","last_change":"2023-12-04T05:39:59.720510+0000","last_active":"2023-12-04T05:40:01.501476+0000","last_peered":"2023-12-04T05:40:01.501476+0000","last_clean":"2023-12-04T05:40:01.501476+0000","last_became_active":"2023-12-04T05:39:59.719704+0000","last_became_peered":"2023-12-04T05:39:59.719704+0000","last_unstale":"2023-12-04T05:40:01.501476+0000","last_undegraded":"2023-12-04T05:40:01.501476+0000","last_fullsized":"2023-12-04T05:40:01.501476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:39:15.714817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681790+0000","last_change":"2023-12-04T05:39:59.711839+0000","last_active":"2023-12-04T05:40:00.681790+0000","last_peered":"2023-12-04T05:40:00.681790+0000","last_clean":"2023-12-04T05:40:00.681790+0000","last_became_active":"2023-12-04T05:39:59.711113+0000","last_became_peered":"2023-12-04T05:39:59.711113+0000","last_unstale":"2023-12-04T05:40:00.681790+0000","last_undegraded":"2023-12-04T05:40:00.681790+0000","last_fullsized":"2023-12-04T05:40:00.681790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:21:17.569525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502449+0000","last_change":"2023-12-04T05:39:58.750554+0000","last_active":"2023-12-04T05:40:01.502449+0000","last_peered":"2023-12-04T05:40:01.502449+0000","last_clean":"2023-12-04T05:40:01.502449+0000","last_became_active":"2023-12-04T05:39:58.750227+0000","last_became_peered":"2023-12-04T05:39:58.750227+0000","last_unstale":"2023-12-04T05:40:01.502449+0000","last_undegraded":"2023-12-04T05:40:01.502449+0000","last_fullsized":"2023-12-04T05:40:01.502449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:13:23.997255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502276+0000","last_change":"2023-12-04T05:39:59.731582+0000","last_active":"2023-12-04T05:40:01.502276+0000","last_peered":"2023-12-04T05:40:01.502276+0000","last_clean":"2023-12-04T05:40:01.502276+0000","last_became_active":"2023-12-04T05:39:59.730822+0000","last_became_peered":"2023-12-04T05:39:59.730822+0000","last_unstale":"2023-12-04T05:40:01.502276+0000","last_undegraded":"2023-12-04T05:40:01.502276+0000","last_fullsized":"2023-12-04T05:40:01.502276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:10:55.996104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501262+0000","last_change":"2023-12-04T05:39:58.740011+0000","last_active":"2023-12-04T05:40:01.501262+0000","last_peered":"2023-12-04T05:40:01.501262+0000","last_clean":"2023-12-04T05:40:01.501262+0000","last_became_active":"2023-12-04T05:39:58.739395+0000","last_became_peered":"2023-12-04T05:39:58.739395+0000","last_unstale":"2023-12-04T05:40:01.501262+0000","last_undegraded":"2023-12-04T05:40:01.501262+0000","last_fullsized":"2023-12-04T05:40:01.501262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:17:48.107311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502182+0000","last_change":"2023-12-04T05:39:58.730186+0000","last_active":"2023-12-04T05:40:01.502182+0000","last_peered":"2023-12-04T05:40:01.502182+0000","last_clean":"2023-12-04T05:40:01.502182+0000","last_became_active":"2023-12-04T05:39:58.729411+0000","last_became_peered":"2023-12-04T05:39:58.729411+0000","last_unstale":"2023-12-04T05:40:01.502182+0000","last_undegraded":"2023-12-04T05:40:01.502182+0000","last_fullsized":"2023-12-04T05:40:01.502182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:39:05.463010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502530+0000","last_change":"2023-12-04T05:39:59.715596+0000","last_active":"2023-12-04T05:40:01.502530+0000","last_peered":"2023-12-04T05:40:01.502530+0000","last_clean":"2023-12-04T05:40:01.502530+0000","last_became_active":"2023-12-04T05:39:59.714842+0000","last_became_peered":"2023-12-04T05:39:59.714842+0000","last_unstale":"2023-12-04T05:40:01.502530+0000","last_undegraded":"2023-12-04T05:40:01.502530+0000","last_fullsized":"2023-12-04T05:40:01.502530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:51:10.874053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682760+0000","last_change":"2023-12-04T05:39:58.722113+0000","last_active":"2023-12-04T05:40:00.682760+0000","last_peered":"2023-12-04T05:40:00.682760+0000","last_clean":"2023-12-04T05:40:00.682760+0000","last_became_active":"2023-12-04T05:39:58.721346+0000","last_became_peered":"2023-12-04T05:39:58.721346+0000","last_unstale":"2023-12-04T05:40:00.682760+0000","last_undegraded":"2023-12-04T05:40:00.682760+0000","last_fullsized":"2023-12-04T05:40:00.682760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:30:11.578047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908461+0000","last_change":"2023-12-04T05:39:59.708575+0000","last_active":"2023-12-04T05:40:00.908461+0000","last_peered":"2023-12-04T05:40:00.908461+0000","last_clean":"2023-12-04T05:40:00.908461+0000","last_became_active":"2023-12-04T05:39:59.707753+0000","last_became_peered":"2023-12-04T05:39:59.707753+0000","last_unstale":"2023-12-04T05:40:00.908461+0000","last_undegraded":"2023-12-04T05:40:00.908461+0000","last_fullsized":"2023-12-04T05:40:00.908461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:47:58.349368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501888+0000","last_change":"2023-12-04T05:39:58.734739+0000","last_active":"2023-12-04T05:40:01.501888+0000","last_peered":"2023-12-04T05:40:01.501888+0000","last_clean":"2023-12-04T05:40:01.501888+0000","last_became_active":"2023-12-04T05:39:58.733897+0000","last_became_peered":"2023-12-04T05:39:58.733897+0000","last_unstale":"2023-12-04T05:40:01.501888+0000","last_undegraded":"2023-12-04T05:40:01.501888+0000","last_fullsized":"2023-12-04T05:40:01.501888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:52:15.157364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683392+0000","last_change":"2023-12-04T05:39:59.715796+0000","last_active":"2023-12-04T05:40:00.683392+0000","last_peered":"2023-12-04T05:40:00.683392+0000","last_clean":"2023-12-04T05:40:00.683392+0000","last_became_active":"2023-12-04T05:39:59.714951+0000","last_became_peered":"2023-12-04T05:39:59.714951+0000","last_unstale":"2023-12-04T05:40:00.683392+0000","last_undegraded":"2023-12-04T05:40:00.683392+0000","last_fullsized":"2023-12-04T05:40:00.683392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:31:12.475136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683603+0000","last_change":"2023-12-04T05:39:58.725387+0000","last_active":"2023-12-04T05:40:00.683603+0000","last_peered":"2023-12-04T05:40:00.683603+0000","last_clean":"2023-12-04T05:40:00.683603+0000","last_became_active":"2023-12-04T05:39:58.724672+0000","last_became_peered":"2023-12-04T05:39:58.724672+0000","last_unstale":"2023-12-04T05:40:00.683603+0000","last_undegraded":"2023-12-04T05:40:00.683603+0000","last_fullsized":"2023-12-04T05:40:00.683603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:19:43.112172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503316+0000","last_change":"2023-12-04T05:39:59.718789+0000","last_active":"2023-12-04T05:40:01.503316+0000","last_peered":"2023-12-04T05:40:01.503316+0000","last_clean":"2023-12-04T05:40:01.503316+0000","last_became_active":"2023-12-04T05:39:59.717927+0000","last_became_peered":"2023-12-04T05:39:59.717927+0000","last_unstale":"2023-12-04T05:40:01.503316+0000","last_undegraded":"2023-12-04T05:40:01.503316+0000","last_fullsized":"2023-12-04T05:40:01.503316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:43:08.771461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.909015+0000","last_change":"2023-12-04T05:39:58.731710+0000","last_active":"2023-12-04T05:40:00.909015+0000","last_peered":"2023-12-04T05:40:00.909015+0000","last_clean":"2023-12-04T05:40:00.909015+0000","last_became_active":"2023-12-04T05:39:58.731373+0000","last_became_peered":"2023-12-04T05:39:58.731373+0000","last_unstale":"2023-12-04T05:40:00.909015+0000","last_undegraded":"2023-12-04T05:40:00.909015+0000","last_fullsized":"2023-12-04T05:40:00.909015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:03:37.899195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682594+0000","last_change":"2023-12-04T05:39:59.725444+0000","last_active":"2023-12-04T05:40:00.682594+0000","last_peered":"2023-12-04T05:40:00.682594+0000","last_clean":"2023-12-04T05:40:00.682594+0000","last_became_active":"2023-12-04T05:39:59.724620+0000","last_became_peered":"2023-12-04T05:39:59.724620+0000","last_unstale":"2023-12-04T05:40:00.682594+0000","last_undegraded":"2023-12-04T05:40:00.682594+0000","last_fullsized":"2023-12-04T05:40:00.682594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:11:17.092060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501523+0000","last_change":"2023-12-04T05:39:58.730135+0000","last_active":"2023-12-04T05:40:01.501523+0000","last_peered":"2023-12-04T05:40:01.501523+0000","last_clean":"2023-12-04T05:40:01.501523+0000","last_became_active":"2023-12-04T05:39:58.729295+0000","last_became_peered":"2023-12-04T05:39:58.729295+0000","last_unstale":"2023-12-04T05:40:01.501523+0000","last_undegraded":"2023-12-04T05:40:01.501523+0000","last_fullsized":"2023-12-04T05:40:01.501523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:24:42.912063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502658+0000","last_change":"2023-12-04T05:39:59.731327+0000","last_active":"2023-12-04T05:40:01.502658+0000","last_peered":"2023-12-04T05:40:01.502658+0000","last_clean":"2023-12-04T05:40:01.502658+0000","last_became_active":"2023-12-04T05:39:59.730427+0000","last_became_peered":"2023-12-04T05:39:59.730427+0000","last_unstale":"2023-12-04T05:40:01.502658+0000","last_undegraded":"2023-12-04T05:40:01.502658+0000","last_fullsized":"2023-12-04T05:40:01.502658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:34:45.211951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502490+0000","last_change":"2023-12-04T05:39:58.711690+0000","last_active":"2023-12-04T05:40:01.502490+0000","last_peered":"2023-12-04T05:40:01.502490+0000","last_clean":"2023-12-04T05:40:01.502490+0000","last_became_active":"2023-12-04T05:39:58.711305+0000","last_became_peered":"2023-12-04T05:39:58.711305+0000","last_unstale":"2023-12-04T05:40:01.502490+0000","last_undegraded":"2023-12-04T05:40:01.502490+0000","last_fullsized":"2023-12-04T05:40:01.502490+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:33:02.159265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501048+0000","last_change":"2023-12-04T05:39:59.702650+0000","last_active":"2023-12-04T05:40:01.501048+0000","last_peered":"2023-12-04T05:40:01.501048+0000","last_clean":"2023-12-04T05:40:01.501048+0000","last_became_active":"2023-12-04T05:39:59.701827+0000","last_became_peered":"2023-12-04T05:39:59.701827+0000","last_unstale":"2023-12-04T05:40:01.501048+0000","last_undegraded":"2023-12-04T05:40:01.501048+0000","last_fullsized":"2023-12-04T05:40:01.501048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:46:45.006062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683932+0000","last_change":"2023-12-04T05:39:58.729371+0000","last_active":"2023-12-04T05:40:00.683932+0000","last_peered":"2023-12-04T05:40:00.683932+0000","last_clean":"2023-12-04T05:40:00.683932+0000","last_became_active":"2023-12-04T05:39:58.727781+0000","last_became_peered":"2023-12-04T05:39:58.727781+0000","last_unstale":"2023-12-04T05:40:00.683932+0000","last_undegraded":"2023-12-04T05:40:00.683932+0000","last_fullsized":"2023-12-04T05:40:00.683932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:37:29.707707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502646+0000","last_change":"2023-12-04T05:39:59.742080+0000","last_active":"2023-12-04T05:40:01.502646+0000","last_peered":"2023-12-04T05:40:01.502646+0000","last_clean":"2023-12-04T05:40:01.502646+0000","last_became_active":"2023-12-04T05:39:59.741440+0000","last_became_peered":"2023-12-04T05:39:59.741440+0000","last_unstale":"2023-12-04T05:40:01.502646+0000","last_undegraded":"2023-12-04T05:40:01.502646+0000","last_fullsized":"2023-12-04T05:40:01.502646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:29:21.379583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.500870+0000","last_change":"2023-12-04T05:39:58.737926+0000","last_active":"2023-12-04T05:40:01.500870+0000","last_peered":"2023-12-04T05:40:01.500870+0000","last_clean":"2023-12-04T05:40:01.500870+0000","last_became_active":"2023-12-04T05:39:58.737403+0000","last_became_peered":"2023-12-04T05:39:58.737403+0000","last_unstale":"2023-12-04T05:40:01.500870+0000","last_undegraded":"2023-12-04T05:40:01.500870+0000","last_fullsized":"2023-12-04T05:40:01.500870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:10:59.035282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501005+0000","last_change":"2023-12-04T05:39:59.710310+0000","last_active":"2023-12-04T05:40:01.501005+0000","last_peered":"2023-12-04T05:40:01.501005+0000","last_clean":"2023-12-04T05:40:01.501005+0000","last_became_active":"2023-12-04T05:39:59.709497+0000","last_became_peered":"2023-12-04T05:39:59.709497+0000","last_unstale":"2023-12-04T05:40:01.501005+0000","last_undegraded":"2023-12-04T05:40:01.501005+0000","last_fullsized":"2023-12-04T05:40:01.501005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:43:43.708311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.678550+0000","last_change":"2023-12-04T05:39:58.728046+0000","last_active":"2023-12-04T05:40:00.678550+0000","last_peered":"2023-12-04T05:40:00.678550+0000","last_clean":"2023-12-04T05:40:00.678550+0000","last_became_active":"2023-12-04T05:39:58.726747+0000","last_became_peered":"2023-12-04T05:39:58.726747+0000","last_unstale":"2023-12-04T05:40:00.678550+0000","last_undegraded":"2023-12-04T05:40:00.678550+0000","last_fullsized":"2023-12-04T05:40:00.678550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:45:11.024395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501689+0000","last_change":"2023-12-04T05:39:59.741770+0000","last_active":"2023-12-04T05:40:01.501689+0000","last_peered":"2023-12-04T05:40:01.501689+0000","last_clean":"2023-12-04T05:40:01.501689+0000","last_became_active":"2023-12-04T05:39:59.741091+0000","last_became_peered":"2023-12-04T05:39:59.741091+0000","last_unstale":"2023-12-04T05:40:01.501689+0000","last_undegraded":"2023-12-04T05:40:01.501689+0000","last_fullsized":"2023-12-04T05:40:01.501689+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:59:20.747612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682211+0000","last_change":"2023-12-04T05:39:58.712970+0000","last_active":"2023-12-04T05:40:00.682211+0000","last_peered":"2023-12-04T05:40:00.682211+0000","last_clean":"2023-12-04T05:40:00.682211+0000","last_became_active":"2023-12-04T05:39:58.711759+0000","last_became_peered":"2023-12-04T05:39:58.711759+0000","last_unstale":"2023-12-04T05:40:00.682211+0000","last_undegraded":"2023-12-04T05:40:00.682211+0000","last_fullsized":"2023-12-04T05:40:00.682211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:02:00.957563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503407+0000","last_change":"2023-12-04T05:39:59.742244+0000","last_active":"2023-12-04T05:40:01.503407+0000","last_peered":"2023-12-04T05:40:01.503407+0000","last_clean":"2023-12-04T05:40:01.503407+0000","last_became_active":"2023-12-04T05:39:59.741379+0000","last_became_peered":"2023-12-04T05:39:59.741379+0000","last_unstale":"2023-12-04T05:40:01.503407+0000","last_undegraded":"2023-12-04T05:40:01.503407+0000","last_fullsized":"2023-12-04T05:40:01.503407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:07:57.685895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.679104+0000","last_change":"2023-12-04T05:39:58.729723+0000","last_active":"2023-12-04T05:40:00.679104+0000","last_peered":"2023-12-04T05:40:00.679104+0000","last_clean":"2023-12-04T05:40:00.679104+0000","last_became_active":"2023-12-04T05:39:58.729410+0000","last_became_peered":"2023-12-04T05:39:58.729410+0000","last_unstale":"2023-12-04T05:40:00.679104+0000","last_undegraded":"2023-12-04T05:40:00.679104+0000","last_fullsized":"2023-12-04T05:40:00.679104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:59:10.799774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503914+0000","last_change":"2023-12-04T05:39:59.741718+0000","last_active":"2023-12-04T05:40:01.503914+0000","last_peered":"2023-12-04T05:40:01.503914+0000","last_clean":"2023-12-04T05:40:01.503914+0000","last_became_active":"2023-12-04T05:39:59.741101+0000","last_became_peered":"2023-12-04T05:39:59.741101+0000","last_unstale":"2023-12-04T05:40:01.503914+0000","last_undegraded":"2023-12-04T05:40:01.503914+0000","last_fullsized":"2023-12-04T05:40:01.503914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:41:03.969161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.909154+0000","last_change":"2023-12-04T05:39:58.737224+0000","last_active":"2023-12-04T05:40:00.909154+0000","last_peered":"2023-12-04T05:40:00.909154+0000","last_clean":"2023-12-04T05:40:00.909154+0000","last_became_active":"2023-12-04T05:39:58.736753+0000","last_became_peered":"2023-12-04T05:39:58.736753+0000","last_unstale":"2023-12-04T05:40:00.909154+0000","last_undegraded":"2023-12-04T05:40:00.909154+0000","last_fullsized":"2023-12-04T05:40:00.909154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:33:34.027796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683405+0000","last_change":"2023-12-04T05:39:59.726776+0000","last_active":"2023-12-04T05:40:00.683405+0000","last_peered":"2023-12-04T05:40:00.683405+0000","last_clean":"2023-12-04T05:40:00.683405+0000","last_became_active":"2023-12-04T05:39:59.725991+0000","last_became_peered":"2023-12-04T05:39:59.725991+0000","last_unstale":"2023-12-04T05:40:00.683405+0000","last_undegraded":"2023-12-04T05:40:00.683405+0000","last_fullsized":"2023-12-04T05:40:00.683405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:36:02.168898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.678822+0000","last_change":"2023-12-04T05:39:58.732942+0000","last_active":"2023-12-04T05:40:00.678822+0000","last_peered":"2023-12-04T05:40:00.678822+0000","last_clean":"2023-12-04T05:40:00.678822+0000","last_became_active":"2023-12-04T05:39:58.732401+0000","last_became_peered":"2023-12-04T05:39:58.732401+0000","last_unstale":"2023-12-04T05:40:00.678822+0000","last_undegraded":"2023-12-04T05:40:00.678822+0000","last_fullsized":"2023-12-04T05:40:00.678822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:23:13.676061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503215+0000","last_change":"2023-12-04T05:39:59.730768+0000","last_active":"2023-12-04T05:40:01.503215+0000","last_peered":"2023-12-04T05:40:01.503215+0000","last_clean":"2023-12-04T05:40:01.503215+0000","last_became_active":"2023-12-04T05:39:59.730207+0000","last_became_peered":"2023-12-04T05:39:59.730207+0000","last_unstale":"2023-12-04T05:40:01.503215+0000","last_undegraded":"2023-12-04T05:40:01.503215+0000","last_fullsized":"2023-12-04T05:40:01.503215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:25:19.473188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T05:39:59.705581+0000","last_change":"2023-12-04T05:39:49.314208+0000","last_active":"2023-12-04T05:39:59.705581+0000","last_peered":"2023-12-04T05:39:59.705581+0000","last_clean":"2023-12-04T05:39:59.705581+0000","last_became_active":"2023-12-04T05:39:49.312262+0000","last_became_peered":"2023-12-04T05:39:49.312262+0000","last_unstale":"2023-12-04T05:39:59.705581+0000","last_undegraded":"2023-12-04T05:39:59.705581+0000","last_fullsized":"2023-12-04T05:39:59.705581+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:48.242612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:48.242612+0000","last_clean_scrub_stamp":"2023-12-04T05:39:48.242612+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:19:26.511345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682745+0000","last_change":"2023-12-04T05:39:58.725538+0000","last_active":"2023-12-04T05:40:00.682745+0000","last_peered":"2023-12-04T05:40:00.682745+0000","last_clean":"2023-12-04T05:40:00.682745+0000","last_became_active":"2023-12-04T05:39:58.724839+0000","last_became_peered":"2023-12-04T05:39:58.724839+0000","last_unstale":"2023-12-04T05:40:00.682745+0000","last_undegraded":"2023-12-04T05:40:00.682745+0000","last_fullsized":"2023-12-04T05:40:00.682745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:34:48.521117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501800+0000","last_change":"2023-12-04T05:39:59.719475+0000","last_active":"2023-12-04T05:40:01.501800+0000","last_peered":"2023-12-04T05:40:01.501800+0000","last_clean":"2023-12-04T05:40:01.501800+0000","last_became_active":"2023-12-04T05:39:59.718727+0000","last_became_peered":"2023-12-04T05:39:59.718727+0000","last_unstale":"2023-12-04T05:40:01.501800+0000","last_undegraded":"2023-12-04T05:40:01.501800+0000","last_fullsized":"2023-12-04T05:40:01.501800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:53:35.787086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501396+0000","last_change":"2023-12-04T05:39:58.751135+0000","last_active":"2023-12-04T05:40:01.501396+0000","last_peered":"2023-12-04T05:40:01.501396+0000","last_clean":"2023-12-04T05:40:01.501396+0000","last_became_active":"2023-12-04T05:39:58.750144+0000","last_became_peered":"2023-12-04T05:39:58.750144+0000","last_unstale":"2023-12-04T05:40:01.501396+0000","last_undegraded":"2023-12-04T05:40:01.501396+0000","last_fullsized":"2023-12-04T05:40:01.501396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:31:32.599659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.500789+0000","last_change":"2023-12-04T05:39:59.729585+0000","last_active":"2023-12-04T05:40:01.500789+0000","last_peered":"2023-12-04T05:40:01.500789+0000","last_clean":"2023-12-04T05:40:01.500789+0000","last_became_active":"2023-12-04T05:39:59.728381+0000","last_became_peered":"2023-12-04T05:39:59.728381+0000","last_unstale":"2023-12-04T05:40:01.500789+0000","last_undegraded":"2023-12-04T05:40:01.500789+0000","last_fullsized":"2023-12-04T05:40:01.500789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:35:11.920755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.678836+0000","last_change":"2023-12-04T05:39:58.731429+0000","last_active":"2023-12-04T05:40:00.678836+0000","last_peered":"2023-12-04T05:40:00.678836+0000","last_clean":"2023-12-04T05:40:00.678836+0000","last_became_active":"2023-12-04T05:39:58.731049+0000","last_became_peered":"2023-12-04T05:39:58.731049+0000","last_unstale":"2023-12-04T05:40:00.678836+0000","last_undegraded":"2023-12-04T05:40:00.678836+0000","last_fullsized":"2023-12-04T05:40:00.678836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:27:16.863460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908531+0000","last_change":"2023-12-04T05:39:59.701324+0000","last_active":"2023-12-04T05:40:00.908531+0000","last_peered":"2023-12-04T05:40:00.908531+0000","last_clean":"2023-12-04T05:40:00.908531+0000","last_became_active":"2023-12-04T05:39:59.700586+0000","last_became_peered":"2023-12-04T05:39:59.700586+0000","last_unstale":"2023-12-04T05:40:00.908531+0000","last_undegraded":"2023-12-04T05:40:00.908531+0000","last_fullsized":"2023-12-04T05:40:00.908531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:37:17.301688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.684152+0000","last_change":"2023-12-04T05:39:58.729635+0000","last_active":"2023-12-04T05:40:00.684152+0000","last_peered":"2023-12-04T05:40:00.684152+0000","last_clean":"2023-12-04T05:40:00.684152+0000","last_became_active":"2023-12-04T05:39:58.728927+0000","last_became_peered":"2023-12-04T05:39:58.728927+0000","last_unstale":"2023-12-04T05:40:00.684152+0000","last_undegraded":"2023-12-04T05:40:00.684152+0000","last_fullsized":"2023-12-04T05:40:00.684152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:58:46.851065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681790+0000","last_change":"2023-12-04T05:39:59.702840+0000","last_active":"2023-12-04T05:40:00.681790+0000","last_peered":"2023-12-04T05:40:00.681790+0000","last_clean":"2023-12-04T05:40:00.681790+0000","last_became_active":"2023-12-04T05:39:59.701652+0000","last_became_peered":"2023-12-04T05:39:59.701652+0000","last_unstale":"2023-12-04T05:40:00.681790+0000","last_undegraded":"2023-12-04T05:40:00.681790+0000","last_fullsized":"2023-12-04T05:40:00.681790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:44:14.733182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908769+0000","last_change":"2023-12-04T05:39:58.721834+0000","last_active":"2023-12-04T05:40:00.908769+0000","last_peered":"2023-12-04T05:40:00.908769+0000","last_clean":"2023-12-04T05:40:00.908769+0000","last_became_active":"2023-12-04T05:39:58.721451+0000","last_became_peered":"2023-12-04T05:39:58.721451+0000","last_unstale":"2023-12-04T05:40:00.908769+0000","last_undegraded":"2023-12-04T05:40:00.908769+0000","last_fullsized":"2023-12-04T05:40:00.908769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:15:52.065853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.684129+0000","last_change":"2023-12-04T05:39:59.731053+0000","last_active":"2023-12-04T05:40:00.684129+0000","last_peered":"2023-12-04T05:40:00.684129+0000","last_clean":"2023-12-04T05:40:00.684129+0000","last_became_active":"2023-12-04T05:39:59.729732+0000","last_became_peered":"2023-12-04T05:39:59.729732+0000","last_unstale":"2023-12-04T05:40:00.684129+0000","last_undegraded":"2023-12-04T05:40:00.684129+0000","last_fullsized":"2023-12-04T05:40:00.684129+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:57:38.581311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503452+0000","last_change":"2023-12-04T05:39:58.751126+0000","last_active":"2023-12-04T05:40:01.503452+0000","last_peered":"2023-12-04T05:40:01.503452+0000","last_clean":"2023-12-04T05:40:01.503452+0000","last_became_active":"2023-12-04T05:39:58.750496+0000","last_became_peered":"2023-12-04T05:39:58.750496+0000","last_unstale":"2023-12-04T05:40:01.503452+0000","last_undegraded":"2023-12-04T05:40:01.503452+0000","last_fullsized":"2023-12-04T05:40:01.503452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:09:22.333993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908579+0000","last_change":"2023-12-04T05:39:59.700032+0000","last_active":"2023-12-04T05:40:00.908579+0000","last_peered":"2023-12-04T05:40:00.908579+0000","last_clean":"2023-12-04T05:40:00.908579+0000","last_became_active":"2023-12-04T05:39:59.699488+0000","last_became_peered":"2023-12-04T05:39:59.699488+0000","last_unstale":"2023-12-04T05:40:00.908579+0000","last_undegraded":"2023-12-04T05:40:00.908579+0000","last_fullsized":"2023-12-04T05:40:00.908579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:59:57.153486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.909014+0000","last_change":"2023-12-04T05:39:58.723126+0000","last_active":"2023-12-04T05:40:00.909014+0000","last_peered":"2023-12-04T05:40:00.909014+0000","last_clean":"2023-12-04T05:40:00.909014+0000","last_became_active":"2023-12-04T05:39:58.722766+0000","last_became_peered":"2023-12-04T05:39:58.722766+0000","last_unstale":"2023-12-04T05:40:00.909014+0000","last_undegraded":"2023-12-04T05:40:00.909014+0000","last_fullsized":"2023-12-04T05:40:00.909014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:28:47.171345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502918+0000","last_change":"2023-12-04T05:39:59.724653+0000","last_active":"2023-12-04T05:40:01.502918+0000","last_peered":"2023-12-04T05:40:01.502918+0000","last_clean":"2023-12-04T05:40:01.502918+0000","last_became_active":"2023-12-04T05:39:59.723945+0000","last_became_peered":"2023-12-04T05:39:59.723945+0000","last_unstale":"2023-12-04T05:40:01.502918+0000","last_undegraded":"2023-12-04T05:40:01.502918+0000","last_fullsized":"2023-12-04T05:40:01.502918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:13:56.812140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501945+0000","last_change":"2023-12-04T05:39:58.746966+0000","last_active":"2023-12-04T05:40:01.501945+0000","last_peered":"2023-12-04T05:40:01.501945+0000","last_clean":"2023-12-04T05:40:01.501945+0000","last_became_active":"2023-12-04T05:39:58.746261+0000","last_became_peered":"2023-12-04T05:39:58.746261+0000","last_unstale":"2023-12-04T05:40:01.501945+0000","last_undegraded":"2023-12-04T05:40:01.501945+0000","last_fullsized":"2023-12-04T05:40:01.501945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:57:46.899291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503381+0000","last_change":"2023-12-04T05:39:59.713259+0000","last_active":"2023-12-04T05:40:01.503381+0000","last_peered":"2023-12-04T05:40:01.503381+0000","last_clean":"2023-12-04T05:40:01.503381+0000","last_became_active":"2023-12-04T05:39:59.712523+0000","last_became_peered":"2023-12-04T05:39:59.712523+0000","last_unstale":"2023-12-04T05:40:01.503381+0000","last_undegraded":"2023-12-04T05:40:01.503381+0000","last_fullsized":"2023-12-04T05:40:01.503381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:41:10.001755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501174+0000","last_change":"2023-12-04T05:39:58.750282+0000","last_active":"2023-12-04T05:40:01.501174+0000","last_peered":"2023-12-04T05:40:01.501174+0000","last_clean":"2023-12-04T05:40:01.501174+0000","last_became_active":"2023-12-04T05:39:58.749651+0000","last_became_peered":"2023-12-04T05:39:58.749651+0000","last_unstale":"2023-12-04T05:40:01.501174+0000","last_undegraded":"2023-12-04T05:40:01.501174+0000","last_fullsized":"2023-12-04T05:40:01.501174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:40:55.086649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681687+0000","last_change":"2023-12-04T05:39:59.704667+0000","last_active":"2023-12-04T05:40:00.681687+0000","last_peered":"2023-12-04T05:40:00.681687+0000","last_clean":"2023-12-04T05:40:00.681687+0000","last_became_active":"2023-12-04T05:39:59.703943+0000","last_became_peered":"2023-12-04T05:39:59.703943+0000","last_unstale":"2023-12-04T05:40:00.681687+0000","last_undegraded":"2023-12-04T05:40:00.681687+0000","last_fullsized":"2023-12-04T05:40:00.681687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:31:53.890532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908698+0000","last_change":"2023-12-04T05:39:58.745277+0000","last_active":"2023-12-04T05:40:00.908698+0000","last_peered":"2023-12-04T05:40:00.908698+0000","last_clean":"2023-12-04T05:40:00.908698+0000","last_became_active":"2023-12-04T05:39:58.744842+0000","last_became_peered":"2023-12-04T05:39:58.744842+0000","last_unstale":"2023-12-04T05:40:00.908698+0000","last_undegraded":"2023-12-04T05:40:00.908698+0000","last_fullsized":"2023-12-04T05:40:00.908698+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:50:21.084855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502964+0000","last_change":"2023-12-04T05:39:59.714217+0000","last_active":"2023-12-04T05:40:01.502964+0000","last_peered":"2023-12-04T05:40:01.502964+0000","last_clean":"2023-12-04T05:40:01.502964+0000","last_became_active":"2023-12-04T05:39:59.713438+0000","last_became_peered":"2023-12-04T05:39:59.713438+0000","last_unstale":"2023-12-04T05:40:01.502964+0000","last_undegraded":"2023-12-04T05:40:01.502964+0000","last_fullsized":"2023-12-04T05:40:01.502964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:37:30.790544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908612+0000","last_change":"2023-12-04T05:39:58.722048+0000","last_active":"2023-12-04T05:40:00.908612+0000","last_peered":"2023-12-04T05:40:00.908612+0000","last_clean":"2023-12-04T05:40:00.908612+0000","last_became_active":"2023-12-04T05:39:58.721782+0000","last_became_peered":"2023-12-04T05:39:58.721782+0000","last_unstale":"2023-12-04T05:40:00.908612+0000","last_undegraded":"2023-12-04T05:40:00.908612+0000","last_fullsized":"2023-12-04T05:40:00.908612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:38:16.023373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502222+0000","last_change":"2023-12-04T05:39:59.715313+0000","last_active":"2023-12-04T05:40:01.502222+0000","last_peered":"2023-12-04T05:40:01.502222+0000","last_clean":"2023-12-04T05:40:01.502222+0000","last_became_active":"2023-12-04T05:39:59.714430+0000","last_became_peered":"2023-12-04T05:39:59.714430+0000","last_unstale":"2023-12-04T05:40:01.502222+0000","last_undegraded":"2023-12-04T05:40:01.502222+0000","last_fullsized":"2023-12-04T05:40:01.502222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:10:01.533077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502795+0000","last_change":"2023-12-04T05:39:58.750921+0000","last_active":"2023-12-04T05:40:01.502795+0000","last_peered":"2023-12-04T05:40:01.502795+0000","last_clean":"2023-12-04T05:40:01.502795+0000","last_became_active":"2023-12-04T05:39:58.750288+0000","last_became_peered":"2023-12-04T05:39:58.750288+0000","last_unstale":"2023-12-04T05:40:01.502795+0000","last_undegraded":"2023-12-04T05:40:01.502795+0000","last_fullsized":"2023-12-04T05:40:01.502795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:04:26.373696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501279+0000","last_change":"2023-12-04T05:39:59.729260+0000","last_active":"2023-12-04T05:40:01.501279+0000","last_peered":"2023-12-04T05:40:01.501279+0000","last_clean":"2023-12-04T05:40:01.501279+0000","last_became_active":"2023-12-04T05:39:59.728639+0000","last_became_peered":"2023-12-04T05:39:59.728639+0000","last_unstale":"2023-12-04T05:40:01.501279+0000","last_undegraded":"2023-12-04T05:40:01.501279+0000","last_fullsized":"2023-12-04T05:40:01.501279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:21:38.039584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683814+0000","last_change":"2023-12-04T05:39:59.731652+0000","last_active":"2023-12-04T05:40:00.683814+0000","last_peered":"2023-12-04T05:40:00.683814+0000","last_clean":"2023-12-04T05:40:00.683814+0000","last_became_active":"2023-12-04T05:39:59.730714+0000","last_became_peered":"2023-12-04T05:39:59.730714+0000","last_unstale":"2023-12-04T05:40:00.683814+0000","last_undegraded":"2023-12-04T05:40:00.683814+0000","last_fullsized":"2023-12-04T05:40:00.683814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:02:27.013221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502732+0000","last_change":"2023-12-04T05:39:58.751530+0000","last_active":"2023-12-04T05:40:01.502732+0000","last_peered":"2023-12-04T05:40:01.502732+0000","last_clean":"2023-12-04T05:40:01.502732+0000","last_became_active":"2023-12-04T05:39:58.750857+0000","last_became_peered":"2023-12-04T05:39:58.750857+0000","last_unstale":"2023-12-04T05:40:01.502732+0000","last_undegraded":"2023-12-04T05:40:01.502732+0000","last_fullsized":"2023-12-04T05:40:01.502732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:43:36.369813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683195+0000","last_change":"2023-12-04T05:39:59.730535+0000","last_active":"2023-12-04T05:40:00.683195+0000","last_peered":"2023-12-04T05:40:00.683195+0000","last_clean":"2023-12-04T05:40:00.683195+0000","last_became_active":"2023-12-04T05:39:59.729717+0000","last_became_peered":"2023-12-04T05:39:59.729717+0000","last_unstale":"2023-12-04T05:40:00.683195+0000","last_undegraded":"2023-12-04T05:40:00.683195+0000","last_fullsized":"2023-12-04T05:40:00.683195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:31:46.120414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502974+0000","last_change":"2023-12-04T05:39:58.752193+0000","last_active":"2023-12-04T05:40:01.502974+0000","last_peered":"2023-12-04T05:40:01.502974+0000","last_clean":"2023-12-04T05:40:01.502974+0000","last_became_active":"2023-12-04T05:39:58.751625+0000","last_became_peered":"2023-12-04T05:39:58.751625+0000","last_unstale":"2023-12-04T05:40:01.502974+0000","last_undegraded":"2023-12-04T05:40:01.502974+0000","last_fullsized":"2023-12-04T05:40:01.502974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:05:05.734494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501167+0000","last_change":"2023-12-04T05:39:59.720197+0000","last_active":"2023-12-04T05:40:01.501167+0000","last_peered":"2023-12-04T05:40:01.501167+0000","last_clean":"2023-12-04T05:40:01.501167+0000","last_became_active":"2023-12-04T05:39:59.718736+0000","last_became_peered":"2023-12-04T05:39:59.718736+0000","last_unstale":"2023-12-04T05:40:01.501167+0000","last_undegraded":"2023-12-04T05:40:01.501167+0000","last_fullsized":"2023-12-04T05:40:01.501167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:41:13.184198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503280+0000","last_change":"2023-12-04T05:39:58.726906+0000","last_active":"2023-12-04T05:40:01.503280+0000","last_peered":"2023-12-04T05:40:01.503280+0000","last_clean":"2023-12-04T05:40:01.503280+0000","last_became_active":"2023-12-04T05:39:58.726415+0000","last_became_peered":"2023-12-04T05:39:58.726415+0000","last_unstale":"2023-12-04T05:40:01.503280+0000","last_undegraded":"2023-12-04T05:40:01.503280+0000","last_fullsized":"2023-12-04T05:40:01.503280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:58:53.715291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503085+0000","last_change":"2023-12-04T05:39:59.711861+0000","last_active":"2023-12-04T05:40:01.503085+0000","last_peered":"2023-12-04T05:40:01.503085+0000","last_clean":"2023-12-04T05:40:01.503085+0000","last_became_active":"2023-12-04T05:39:59.710933+0000","last_became_peered":"2023-12-04T05:39:59.710933+0000","last_unstale":"2023-12-04T05:40:01.503085+0000","last_undegraded":"2023-12-04T05:40:01.503085+0000","last_fullsized":"2023-12-04T05:40:01.503085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:40:46.361768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501520+0000","last_change":"2023-12-04T05:39:58.740704+0000","last_active":"2023-12-04T05:40:01.501520+0000","last_peered":"2023-12-04T05:40:01.501520+0000","last_clean":"2023-12-04T05:40:01.501520+0000","last_became_active":"2023-12-04T05:39:58.740440+0000","last_became_peered":"2023-12-04T05:39:58.740440+0000","last_unstale":"2023-12-04T05:40:01.501520+0000","last_undegraded":"2023-12-04T05:40:01.501520+0000","last_fullsized":"2023-12-04T05:40:01.501520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:20:18.197917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":798720,"data_stored":659649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0589999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0940000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3879999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31380,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340460,"statfs":{"total":96636764160,"available":96604631040,"internally_reserved":0,"allocated":204800,"data_stored":69281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.26}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.163}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0169999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.375}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2270000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":69281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.111}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.052}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.476}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0580000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0960000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1200000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3080000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4399999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":798720,"data_stored":659649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-04T05:40:05.779 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-04T05:40:05.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.927+0000 7efe1405e640 1 Processor -- start 2023-12-04T05:40:05.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.927+0000 7efe1405e640 1 -- start start 2023-12-04T05:40:05.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe1405e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7efe0c071080 0x7efe0c162500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:05.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe1405e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe0c15a390 con 0x7efe0c150dc0 2023-12-04T05:40:05.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe1405e640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe0c157040 con 0x7efe0c156b70 2023-12-04T05:40:05.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe1405e640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe0c157ba0 con 0x7efe0c071080 2023-12-04T05:40:05.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe11dfa640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7efe0c150dc0 0x7efe0c15a820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60322/0 (socket says 172.21.15.107:60322) 2023-12-04T05:40:05.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe11dfa640 1 -- 172.21.15.107:0/4081401713 learned_addr learned my addr 172.21.15.107:0/4081401713 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:05.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe10df8640 1 -- 172.21.15.107:0/4081401713 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901404976 0 0) 0x7efe0c15a390 con 0x7efe0c150dc0 2023-12-04T05:40:05.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe10df8640 1 -- 172.21.15.107:0/4081401713 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdf8005740 con 0x7efe0c150dc0 2023-12-04T05:40:05.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe125fb640 1 --2- 172.21.15.107:0/4081401713 >> v2:172.21.15.195:3300/0 conn(0x7efe0c071080 0x7efe0c162500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:05.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe10df8640 1 -- 172.21.15.107:0/4081401713 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1058647878 0 0) 0x7efdf8005740 con 0x7efe0c150dc0 2023-12-04T05:40:05.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe10df8640 1 -- 172.21.15.107:0/4081401713 >> v2:172.21.15.195:3300/0 conn(0x7efe0c071080 msgr2=0x7efe0c162500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:05.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe10df8640 1 --2- 172.21.15.107:0/4081401713 >> v2:172.21.15.195:3300/0 conn(0x7efe0c071080 0x7efe0c162500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe10df8640 1 -- 172.21.15.107:0/4081401713 >> v1:172.21.15.195:6789/0 conn(0x7efe0c156b70 legacy=0x7efe0c071740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:05.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe10df8640 1 -- 172.21.15.107:0/4081401713 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0c157300 con 0x7efe0c150dc0 2023-12-04T05:40:05.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe10df8640 1 -- 172.21.15.107:0/4081401713 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7efe04001680 con 0x7efe0c150dc0 2023-12-04T05:40:05.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe10df8640 1 -- 172.21.15.107:0/4081401713 >> v1:172.21.15.107:6789/0 conn(0x7efe0c150dc0 legacy=0x7efe0c15a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:05.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe10df8640 1 --2- 172.21.15.107:0/4081401713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdf8005b90 0x7efdf8005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:05.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe10df8640 1 --2- 172.21.15.107:0/4081401713 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efdf80069e0 0x7efdf8006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:05.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe10df8640 1 --2- 172.21.15.107:0/4081401713 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efdf80073a0 0x7efdf800dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:05.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe10df8640 1 -- 172.21.15.107:0/4081401713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c157ba0 con 0x7efdf8005b90 2023-12-04T05:40:05.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe10df8640 1 -- 172.21.15.107:0/4081401713 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf800e2d0 con 0x7efdf80073a0 2023-12-04T05:40:05.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe10df8640 1 -- 172.21.15.107:0/4081401713 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf800e490 con 0x7efdf80069e0 2023-12-04T05:40:05.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe11dfa640 1 --2- 172.21.15.107:0/4081401713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdf8005b90 0x7efdf8005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:05.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe125fb640 1 --2- 172.21.15.107:0/4081401713 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efdf80073a0 0x7efdf800dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:05.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe11dfa640 1 -- 172.21.15.107:0/4081401713 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efdf80069e0 msgr2=0x7efdf8006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:05.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe11dfa640 1 --2- 172.21.15.107:0/4081401713 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efdf80069e0 0x7efdf8006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe11dfa640 1 -- 172.21.15.107:0/4081401713 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efdf80073a0 msgr2=0x7efdf800dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:05.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe11dfa640 1 --2- 172.21.15.107:0/4081401713 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efdf80073a0 0x7efdf800dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe11dfa640 1 -- 172.21.15.107:0/4081401713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf8005740 con 0x7efdf8005b90 2023-12-04T05:40:05.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe125fb640 1 -- 172.21.15.107:0/4081401713 reap_dead start 2023-12-04T05:40:05.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe11dfa640 1 --2- 172.21.15.107:0/4081401713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdf8005b90 0x7efdf8005f90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efe04001ec0 tx=0x7efe0401cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cac18e57a7d53f6 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:05.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe10df8640 1 -- 172.21.15.107:0/4081401713 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe040a5b00 con 0x7efdf8005b90 2023-12-04T05:40:05.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe10df8640 1 -- 172.21.15.107:0/4081401713 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe040a5c80 con 0x7efdf8005b90 2023-12-04T05:40:05.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.931+0000 7efe10df8640 1 -- 172.21.15.107:0/4081401713 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe04005b20 con 0x7efdf8005b90 2023-12-04T05:40:05.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe1405e640 1 -- 172.21.15.107:0/4081401713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdf8005b90 msgr2=0x7efdf8005f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:05.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe1405e640 1 --2- 172.21.15.107:0/4081401713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdf8005b90 0x7efdf8005f90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efe04001ec0 tx=0x7efe0401cc20 comp rx=0 tx=0).stop 2023-12-04T05:40:05.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe1405e640 1 -- 172.21.15.107:0/4081401713 shutdown_connections 2023-12-04T05:40:05.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe1405e640 1 --2- 172.21.15.107:0/4081401713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdf8005b90 0x7efdf8005f90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe1405e640 1 -- 172.21.15.107:0/4081401713 >> 172.21.15.107:0/4081401713 conn(0x7efe0c0cc490 msgr2=0x7efe0c0cc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:05.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe1405e640 1 -- 172.21.15.107:0/4081401713 shutdown_connections 2023-12-04T05:40:05.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe1405e640 1 -- 172.21.15.107:0/4081401713 wait complete. 2023-12-04T05:40:05.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe1405e640 1 Processor -- start 2023-12-04T05:40:05.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe1405e640 1 -- start start 2023-12-04T05:40:05.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe1405e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efdf8005b90 0x7efe0c1e4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:05.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe1405e640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efe0c1585e0 0x7efe0c1e4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:05.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe1405e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe0c158d80 0x7efe0c159200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:05.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe1405e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c1e2b80 con 0x7efe0c158d80 2023-12-04T05:40:05.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe1405e640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c159740 con 0x7efe0c1585e0 2023-12-04T05:40:05.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe1405e640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c157dd0 con 0x7efdf8005b90 2023-12-04T05:40:05.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe125fb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe0c158d80 0x7efe0c159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:05.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe125fb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe0c158d80 0x7efe0c159200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33918/0 (socket says 172.21.15.107:33918) 2023-12-04T05:40:05.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe125fb640 1 -- 172.21.15.107:0/101075595 learned_addr learned my addr 172.21.15.107:0/101075595 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:05.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe11dfa640 1 --2- 172.21.15.107:0/101075595 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efdf8005b90 0x7efe0c1e4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:05.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe115f9640 1 --2- 172.21.15.107:0/101075595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efe0c1585e0 0x7efe0c1e4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:05.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe125fb640 1 -- 172.21.15.107:0/101075595 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efdf8005b90 msgr2=0x7efe0c1e4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:05.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe125fb640 1 --2- 172.21.15.107:0/101075595 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efdf8005b90 0x7efe0c1e4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe125fb640 1 -- 172.21.15.107:0/101075595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efe0c1585e0 msgr2=0x7efe0c1e4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:05.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe125fb640 1 --2- 172.21.15.107:0/101075595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efe0c1585e0 0x7efe0c1e4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:05.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe125fb640 1 -- 172.21.15.107:0/101075595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0c0c44e0 con 0x7efe0c158d80 2023-12-04T05:40:05.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe11dfa640 1 --2- 172.21.15.107:0/101075595 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efdf8005b90 0x7efe0c1e4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:05.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe115f9640 1 --2- 172.21.15.107:0/101075595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efe0c1585e0 0x7efe0c1e4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:05.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe125fb640 1 --2- 172.21.15.107:0/101075595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe0c158d80 0x7efe0c159200 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efe0c154740 tx=0x7efe08009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:05.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe027fc640 1 -- 172.21.15.107:0/101075595 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe08007db0 con 0x7efe0c158d80 2023-12-04T05:40:05.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe027fc640 1 -- 172.21.15.107:0/101075595 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe08014400 con 0x7efe0c158d80 2023-12-04T05:40:05.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe1405e640 1 -- 172.21.15.107:0/101075595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe0c1ef920 con 0x7efe0c158d80 2023-12-04T05:40:05.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe027fc640 1 -- 172.21.15.107:0/101075595 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe080147b0 con 0x7efe0c158d80 2023-12-04T05:40:05.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.935+0000 7efe1405e640 1 -- 172.21.15.107:0/101075595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe0c158370 con 0x7efe0c158d80 2023-12-04T05:40:05.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.939+0000 7efe1405e640 1 -- 172.21.15.107:0/101075595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe0c067ec0 con 0x7efe0c158d80 2023-12-04T05:40:05.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.939+0000 7efe027fc640 1 -- 172.21.15.107:0/101075595 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7efe08014910 con 0x7efe0c158d80 2023-12-04T05:40:05.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.939+0000 7efe027fc640 1 --2- 172.21.15.107:0/101075595 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efdf0066900 0x7efdf0068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:05.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.939+0000 7efe027fc640 1 -- 172.21.15.107:0/101075595 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efe080840b0 con 0x7efe0c158d80 2023-12-04T05:40:05.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.939+0000 7efe11dfa640 1 --2- 172.21.15.107:0/101075595 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efdf0066900 0x7efdf0068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:05.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.939+0000 7efe11dfa640 1 --2- 172.21.15.107:0/101075595 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efdf0066900 0x7efdf0068dc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efe040ab000 tx=0x7efe04006040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:05.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:05.943+0000 7efe027fc640 1 -- 172.21.15.107:0/101075595 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7efe080513a0 con 0x7efe0c158d80 2023-12-04T05:40:06.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.091+0000 7efe1405e640 1 -- 172.21.15.107:0/101075595 --> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efe0c0c3b10 con 0x7efdf0066900 2023-12-04T05:40:06.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.111+0000 7efe027fc640 1 -- 172.21.15.107:0/101075595 <== mgr.4104 v2:172.21.15.107:6832/2693048026 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264484 (secure 0 0 0) 0x7efe0c0c3b10 con 0x7efdf0066900 2023-12-04T05:40:06.118 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:06.120 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-12-04T05:40:06.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.115+0000 7efe1405e640 1 -- 172.21.15.107:0/101075595 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efdf0066900 msgr2=0x7efdf0068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:06.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.115+0000 7efe1405e640 1 --2- 172.21.15.107:0/101075595 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efdf0066900 0x7efdf0068dc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efe040ab000 tx=0x7efe04006040 comp rx=0 tx=0).stop 2023-12-04T05:40:06.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.115+0000 7efe1405e640 1 -- 172.21.15.107:0/101075595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe0c158d80 msgr2=0x7efe0c159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:06.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.115+0000 7efe1405e640 1 --2- 172.21.15.107:0/101075595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe0c158d80 0x7efe0c159200 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efe0c154740 tx=0x7efe08009dc0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.115+0000 7efe1405e640 1 -- 172.21.15.107:0/101075595 shutdown_connections 2023-12-04T05:40:06.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.115+0000 7efe1405e640 1 --2- 172.21.15.107:0/101075595 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efdf0066900 0x7efdf0068dc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.115+0000 7efe1405e640 1 --2- 172.21.15.107:0/101075595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe0c158d80 0x7efe0c159200 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.115+0000 7efe1405e640 1 --2- 172.21.15.107:0/101075595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efe0c1585e0 0x7efe0c1e4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.115+0000 7efe1405e640 1 --2- 172.21.15.107:0/101075595 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efdf8005b90 0x7efe0c1e4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.115+0000 7efe1405e640 1 -- 172.21.15.107:0/101075595 >> 172.21.15.107:0/101075595 conn(0x7efe0c0cc490 msgr2=0x7efe0c0c63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:06.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.115+0000 7efe1405e640 1 -- 172.21.15.107:0/101075595 shutdown_connections 2023-12-04T05:40:06.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.115+0000 7efe1405e640 1 -- 172.21.15.107:0/101075595 wait complete. 2023-12-04T05:40:06.136 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-04T05:40:05.492635+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":86,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235636,"kb_used_data":2808,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739084,"statfs":{"total":773094113280,"available":772852822016,"internally_reserved":0,"allocated":2875392,"data_stored":1760184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":67,"apply_latency_ms":67,"commit_latency_ns":67000000,"apply_latency_ns":67000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.799872"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503577+0000","last_change":"2023-12-04T05:39:59.736570+0000","last_active":"2023-12-04T05:40:01.503577+0000","last_peered":"2023-12-04T05:40:01.503577+0000","last_clean":"2023-12-04T05:40:01.503577+0000","last_became_active":"2023-12-04T05:39:59.736000+0000","last_became_peered":"2023-12-04T05:39:59.736000+0000","last_unstale":"2023-12-04T05:40:01.503577+0000","last_undegraded":"2023-12-04T05:40:01.503577+0000","last_fullsized":"2023-12-04T05:40:01.503577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:34:15.458889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908781+0000","last_change":"2023-12-04T05:39:59.703955+0000","last_active":"2023-12-04T05:40:00.908781+0000","last_peered":"2023-12-04T05:40:00.908781+0000","last_clean":"2023-12-04T05:40:00.908781+0000","last_became_active":"2023-12-04T05:39:59.703032+0000","last_became_peered":"2023-12-04T05:39:59.703032+0000","last_unstale":"2023-12-04T05:40:00.908781+0000","last_undegraded":"2023-12-04T05:40:00.908781+0000","last_fullsized":"2023-12-04T05:40:00.908781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:22:09.010348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501353+0000","last_change":"2023-12-04T05:39:59.705340+0000","last_active":"2023-12-04T05:40:01.501353+0000","last_peered":"2023-12-04T05:40:01.501353+0000","last_clean":"2023-12-04T05:40:01.501353+0000","last_became_active":"2023-12-04T05:39:59.699479+0000","last_became_peered":"2023-12-04T05:39:59.699479+0000","last_unstale":"2023-12-04T05:40:01.501353+0000","last_undegraded":"2023-12-04T05:40:01.501353+0000","last_fullsized":"2023-12-04T05:40:01.501353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:12:00.122168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683148+0000","last_change":"2023-12-04T05:39:59.728698+0000","last_active":"2023-12-04T05:40:00.683148+0000","last_peered":"2023-12-04T05:40:00.683148+0000","last_clean":"2023-12-04T05:40:00.683148+0000","last_became_active":"2023-12-04T05:39:59.727914+0000","last_became_peered":"2023-12-04T05:39:59.727914+0000","last_unstale":"2023-12-04T05:40:00.683148+0000","last_undegraded":"2023-12-04T05:40:00.683148+0000","last_fullsized":"2023-12-04T05:40:00.683148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:39:17.269322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502651+0000","last_change":"2023-12-04T05:39:59.726320+0000","last_active":"2023-12-04T05:40:01.502651+0000","last_peered":"2023-12-04T05:40:01.502651+0000","last_clean":"2023-12-04T05:40:01.502651+0000","last_became_active":"2023-12-04T05:39:59.722627+0000","last_became_peered":"2023-12-04T05:39:59.722627+0000","last_unstale":"2023-12-04T05:40:01.502651+0000","last_undegraded":"2023-12-04T05:40:01.502651+0000","last_fullsized":"2023-12-04T05:40:01.502651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:40:48.940938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682860+0000","last_change":"2023-12-04T05:39:59.718363+0000","last_active":"2023-12-04T05:40:00.682860+0000","last_peered":"2023-12-04T05:40:00.682860+0000","last_clean":"2023-12-04T05:40:00.682860+0000","last_became_active":"2023-12-04T05:39:59.717475+0000","last_became_peered":"2023-12-04T05:39:59.717475+0000","last_unstale":"2023-12-04T05:40:00.682860+0000","last_undegraded":"2023-12-04T05:40:00.682860+0000","last_fullsized":"2023-12-04T05:40:00.682860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:02:56.684095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501353+0000","last_change":"2023-12-04T05:39:59.724035+0000","last_active":"2023-12-04T05:40:01.501353+0000","last_peered":"2023-12-04T05:40:01.501353+0000","last_clean":"2023-12-04T05:40:01.501353+0000","last_became_active":"2023-12-04T05:39:59.723277+0000","last_became_peered":"2023-12-04T05:39:59.723277+0000","last_unstale":"2023-12-04T05:40:01.501353+0000","last_undegraded":"2023-12-04T05:40:01.501353+0000","last_fullsized":"2023-12-04T05:40:01.501353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:34:54.379064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502157+0000","last_change":"2023-12-04T05:39:59.720028+0000","last_active":"2023-12-04T05:40:01.502157+0000","last_peered":"2023-12-04T05:40:01.502157+0000","last_clean":"2023-12-04T05:40:01.502157+0000","last_became_active":"2023-12-04T05:39:59.719236+0000","last_became_peered":"2023-12-04T05:39:59.719236+0000","last_unstale":"2023-12-04T05:40:01.502157+0000","last_undegraded":"2023-12-04T05:40:01.502157+0000","last_fullsized":"2023-12-04T05:40:01.502157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:23:39.909425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503754+0000","last_change":"2023-12-04T05:39:59.732301+0000","last_active":"2023-12-04T05:40:01.503754+0000","last_peered":"2023-12-04T05:40:01.503754+0000","last_clean":"2023-12-04T05:40:01.503754+0000","last_became_active":"2023-12-04T05:39:59.731561+0000","last_became_peered":"2023-12-04T05:39:59.731561+0000","last_unstale":"2023-12-04T05:40:01.503754+0000","last_undegraded":"2023-12-04T05:40:01.503754+0000","last_fullsized":"2023-12-04T05:40:01.503754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:49:40.179867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501688+0000","last_change":"2023-12-04T05:39:59.721633+0000","last_active":"2023-12-04T05:40:01.501688+0000","last_peered":"2023-12-04T05:40:01.501688+0000","last_clean":"2023-12-04T05:40:01.501688+0000","last_became_active":"2023-12-04T05:39:59.720846+0000","last_became_peered":"2023-12-04T05:39:59.720846+0000","last_unstale":"2023-12-04T05:40:01.501688+0000","last_undegraded":"2023-12-04T05:40:01.501688+0000","last_fullsized":"2023-12-04T05:40:01.501688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:44:34.893307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501835+0000","last_change":"2023-12-04T05:39:59.721173+0000","last_active":"2023-12-04T05:40:01.501835+0000","last_peered":"2023-12-04T05:40:01.501835+0000","last_clean":"2023-12-04T05:40:01.501835+0000","last_became_active":"2023-12-04T05:39:59.720371+0000","last_became_peered":"2023-12-04T05:39:59.720371+0000","last_unstale":"2023-12-04T05:40:01.501835+0000","last_undegraded":"2023-12-04T05:40:01.501835+0000","last_fullsized":"2023-12-04T05:40:01.501835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:12:23.683644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682272+0000","last_change":"2023-12-04T05:39:59.710269+0000","last_active":"2023-12-04T05:40:00.682272+0000","last_peered":"2023-12-04T05:40:00.682272+0000","last_clean":"2023-12-04T05:40:00.682272+0000","last_became_active":"2023-12-04T05:39:59.709565+0000","last_became_peered":"2023-12-04T05:39:59.709565+0000","last_unstale":"2023-12-04T05:40:00.682272+0000","last_undegraded":"2023-12-04T05:40:00.682272+0000","last_fullsized":"2023-12-04T05:40:00.682272+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:11:13.222803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501884+0000","last_change":"2023-12-04T05:39:59.710484+0000","last_active":"2023-12-04T05:40:01.501884+0000","last_peered":"2023-12-04T05:40:01.501884+0000","last_clean":"2023-12-04T05:40:01.501884+0000","last_became_active":"2023-12-04T05:39:59.709623+0000","last_became_peered":"2023-12-04T05:39:59.709623+0000","last_unstale":"2023-12-04T05:40:01.501884+0000","last_undegraded":"2023-12-04T05:40:01.501884+0000","last_fullsized":"2023-12-04T05:40:01.501884+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:40:05.852673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.500880+0000","last_change":"2023-12-04T05:39:59.715685+0000","last_active":"2023-12-04T05:40:01.500880+0000","last_peered":"2023-12-04T05:40:01.500880+0000","last_clean":"2023-12-04T05:40:01.500880+0000","last_became_active":"2023-12-04T05:39:59.714890+0000","last_became_peered":"2023-12-04T05:39:59.714890+0000","last_unstale":"2023-12-04T05:40:01.500880+0000","last_undegraded":"2023-12-04T05:40:01.500880+0000","last_fullsized":"2023-12-04T05:40:01.500880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:59:34.243902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908703+0000","last_change":"2023-12-04T05:39:59.688405+0000","last_active":"2023-12-04T05:40:00.908703+0000","last_peered":"2023-12-04T05:40:00.908703+0000","last_clean":"2023-12-04T05:40:00.908703+0000","last_became_active":"2023-12-04T05:39:59.687603+0000","last_became_peered":"2023-12-04T05:39:59.687603+0000","last_unstale":"2023-12-04T05:40:00.908703+0000","last_undegraded":"2023-12-04T05:40:00.908703+0000","last_fullsized":"2023-12-04T05:40:00.908703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:44:51.269794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681203+0000","last_change":"2023-12-04T05:39:59.718543+0000","last_active":"2023-12-04T05:40:00.681203+0000","last_peered":"2023-12-04T05:40:00.681203+0000","last_clean":"2023-12-04T05:40:00.681203+0000","last_became_active":"2023-12-04T05:39:59.717826+0000","last_became_peered":"2023-12-04T05:39:59.717826+0000","last_unstale":"2023-12-04T05:40:00.681203+0000","last_undegraded":"2023-12-04T05:40:00.681203+0000","last_fullsized":"2023-12-04T05:40:00.681203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:31:03.248114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.684087+0000","last_change":"2023-12-04T05:39:59.724278+0000","last_active":"2023-12-04T05:40:00.684087+0000","last_peered":"2023-12-04T05:40:00.684087+0000","last_clean":"2023-12-04T05:40:00.684087+0000","last_became_active":"2023-12-04T05:39:59.723384+0000","last_became_peered":"2023-12-04T05:39:59.723384+0000","last_unstale":"2023-12-04T05:40:00.684087+0000","last_undegraded":"2023-12-04T05:40:00.684087+0000","last_fullsized":"2023-12-04T05:40:00.684087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:07:20.576260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.679022+0000","last_change":"2023-12-04T05:39:59.711802+0000","last_active":"2023-12-04T05:40:00.679022+0000","last_peered":"2023-12-04T05:40:00.679022+0000","last_clean":"2023-12-04T05:40:00.679022+0000","last_became_active":"2023-12-04T05:39:59.711044+0000","last_became_peered":"2023-12-04T05:39:59.711044+0000","last_unstale":"2023-12-04T05:40:00.679022+0000","last_undegraded":"2023-12-04T05:40:00.679022+0000","last_fullsized":"2023-12-04T05:40:00.679022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:42:28.221010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503300+0000","last_change":"2023-12-04T05:39:59.725226+0000","last_active":"2023-12-04T05:40:01.503300+0000","last_peered":"2023-12-04T05:40:01.503300+0000","last_clean":"2023-12-04T05:40:01.503300+0000","last_became_active":"2023-12-04T05:39:59.724334+0000","last_became_peered":"2023-12-04T05:39:59.724334+0000","last_unstale":"2023-12-04T05:40:01.503300+0000","last_undegraded":"2023-12-04T05:40:01.503300+0000","last_fullsized":"2023-12-04T05:40:01.503300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:48:30.439285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501624+0000","last_change":"2023-12-04T05:39:59.729858+0000","last_active":"2023-12-04T05:40:01.501624+0000","last_peered":"2023-12-04T05:40:01.501624+0000","last_clean":"2023-12-04T05:40:01.501624+0000","last_became_active":"2023-12-04T05:39:59.728989+0000","last_became_peered":"2023-12-04T05:39:59.728989+0000","last_unstale":"2023-12-04T05:40:01.501624+0000","last_undegraded":"2023-12-04T05:40:01.501624+0000","last_fullsized":"2023-12-04T05:40:01.501624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:42:01.099743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501616+0000","last_change":"2023-12-04T05:39:59.724022+0000","last_active":"2023-12-04T05:40:01.501616+0000","last_peered":"2023-12-04T05:40:01.501616+0000","last_clean":"2023-12-04T05:40:01.501616+0000","last_became_active":"2023-12-04T05:39:59.723120+0000","last_became_peered":"2023-12-04T05:39:59.723120+0000","last_unstale":"2023-12-04T05:40:01.501616+0000","last_undegraded":"2023-12-04T05:40:01.501616+0000","last_fullsized":"2023-12-04T05:40:01.501616+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:13:50.048757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683322+0000","last_change":"2023-12-04T05:39:59.714806+0000","last_active":"2023-12-04T05:40:00.683322+0000","last_peered":"2023-12-04T05:40:00.683322+0000","last_clean":"2023-12-04T05:40:00.683322+0000","last_became_active":"2023-12-04T05:39:59.713917+0000","last_became_peered":"2023-12-04T05:39:59.713917+0000","last_unstale":"2023-12-04T05:40:00.683322+0000","last_undegraded":"2023-12-04T05:40:00.683322+0000","last_fullsized":"2023-12-04T05:40:00.683322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:18:03.128562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501966+0000","last_change":"2023-12-04T05:39:59.729098+0000","last_active":"2023-12-04T05:40:01.501966+0000","last_peered":"2023-12-04T05:40:01.501966+0000","last_clean":"2023-12-04T05:40:01.501966+0000","last_became_active":"2023-12-04T05:39:59.728449+0000","last_became_peered":"2023-12-04T05:39:59.728449+0000","last_unstale":"2023-12-04T05:40:01.501966+0000","last_undegraded":"2023-12-04T05:40:01.501966+0000","last_fullsized":"2023-12-04T05:40:01.501966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:25:12.098431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682551+0000","last_change":"2023-12-04T05:39:59.708300+0000","last_active":"2023-12-04T05:40:00.682551+0000","last_peered":"2023-12-04T05:40:00.682551+0000","last_clean":"2023-12-04T05:40:00.682551+0000","last_became_active":"2023-12-04T05:39:59.707530+0000","last_became_peered":"2023-12-04T05:39:59.707530+0000","last_unstale":"2023-12-04T05:40:00.682551+0000","last_undegraded":"2023-12-04T05:40:00.682551+0000","last_fullsized":"2023-12-04T05:40:00.682551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:11:40.053312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502562+0000","last_change":"2023-12-04T05:39:59.729406+0000","last_active":"2023-12-04T05:40:01.502562+0000","last_peered":"2023-12-04T05:40:01.502562+0000","last_clean":"2023-12-04T05:40:01.502562+0000","last_became_active":"2023-12-04T05:39:59.728677+0000","last_became_peered":"2023-12-04T05:39:59.728677+0000","last_unstale":"2023-12-04T05:40:01.502562+0000","last_undegraded":"2023-12-04T05:40:01.502562+0000","last_fullsized":"2023-12-04T05:40:01.502562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:03:26.954387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501922+0000","last_change":"2023-12-04T05:39:59.733628+0000","last_active":"2023-12-04T05:40:01.501922+0000","last_peered":"2023-12-04T05:40:01.501922+0000","last_clean":"2023-12-04T05:40:01.501922+0000","last_became_active":"2023-12-04T05:39:59.726137+0000","last_became_peered":"2023-12-04T05:39:59.726137+0000","last_unstale":"2023-12-04T05:40:01.501922+0000","last_undegraded":"2023-12-04T05:40:01.501922+0000","last_fullsized":"2023-12-04T05:40:01.501922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:49:17.630085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908788+0000","last_change":"2023-12-04T05:39:59.727169+0000","last_active":"2023-12-04T05:40:00.908788+0000","last_peered":"2023-12-04T05:40:00.908788+0000","last_clean":"2023-12-04T05:40:00.908788+0000","last_became_active":"2023-12-04T05:39:59.725378+0000","last_became_peered":"2023-12-04T05:39:59.725378+0000","last_unstale":"2023-12-04T05:40:00.908788+0000","last_undegraded":"2023-12-04T05:40:00.908788+0000","last_fullsized":"2023-12-04T05:40:00.908788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:23:21.007397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502427+0000","last_change":"2023-12-04T05:39:59.729516+0000","last_active":"2023-12-04T05:40:01.502427+0000","last_peered":"2023-12-04T05:40:01.502427+0000","last_clean":"2023-12-04T05:40:01.502427+0000","last_became_active":"2023-12-04T05:39:59.728835+0000","last_became_peered":"2023-12-04T05:39:59.728835+0000","last_unstale":"2023-12-04T05:40:01.502427+0000","last_undegraded":"2023-12-04T05:40:01.502427+0000","last_fullsized":"2023-12-04T05:40:01.502427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:37:37.490362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501933+0000","last_change":"2023-12-04T05:39:59.724592+0000","last_active":"2023-12-04T05:40:01.501933+0000","last_peered":"2023-12-04T05:40:01.501933+0000","last_clean":"2023-12-04T05:40:01.501933+0000","last_became_active":"2023-12-04T05:39:59.723876+0000","last_became_peered":"2023-12-04T05:39:59.723876+0000","last_unstale":"2023-12-04T05:40:01.501933+0000","last_undegraded":"2023-12-04T05:40:01.501933+0000","last_fullsized":"2023-12-04T05:40:01.501933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:09:26.625728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683703+0000","last_change":"2023-12-04T05:39:59.726540+0000","last_active":"2023-12-04T05:40:00.683703+0000","last_peered":"2023-12-04T05:40:00.683703+0000","last_clean":"2023-12-04T05:40:00.683703+0000","last_became_active":"2023-12-04T05:39:59.725781+0000","last_became_peered":"2023-12-04T05:39:59.725781+0000","last_unstale":"2023-12-04T05:40:00.683703+0000","last_undegraded":"2023-12-04T05:40:00.683703+0000","last_fullsized":"2023-12-04T05:40:00.683703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:59:22.013410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.678751+0000","last_change":"2023-12-04T05:39:59.708376+0000","last_active":"2023-12-04T05:40:00.678751+0000","last_peered":"2023-12-04T05:40:00.678751+0000","last_clean":"2023-12-04T05:40:00.678751+0000","last_became_active":"2023-12-04T05:39:59.707604+0000","last_became_peered":"2023-12-04T05:39:59.707604+0000","last_unstale":"2023-12-04T05:40:00.678751+0000","last_undegraded":"2023-12-04T05:40:00.678751+0000","last_fullsized":"2023-12-04T05:40:00.678751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:00:23.385619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501476+0000","last_change":"2023-12-04T05:39:59.720510+0000","last_active":"2023-12-04T05:40:01.501476+0000","last_peered":"2023-12-04T05:40:01.501476+0000","last_clean":"2023-12-04T05:40:01.501476+0000","last_became_active":"2023-12-04T05:39:59.719704+0000","last_became_peered":"2023-12-04T05:39:59.719704+0000","last_unstale":"2023-12-04T05:40:01.501476+0000","last_undegraded":"2023-12-04T05:40:01.501476+0000","last_fullsized":"2023-12-04T05:40:01.501476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:39:15.714817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681790+0000","last_change":"2023-12-04T05:39:59.711839+0000","last_active":"2023-12-04T05:40:00.681790+0000","last_peered":"2023-12-04T05:40:00.681790+0000","last_clean":"2023-12-04T05:40:00.681790+0000","last_became_active":"2023-12-04T05:39:59.711113+0000","last_became_peered":"2023-12-04T05:39:59.711113+0000","last_unstale":"2023-12-04T05:40:00.681790+0000","last_undegraded":"2023-12-04T05:40:00.681790+0000","last_fullsized":"2023-12-04T05:40:00.681790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:21:17.569525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502449+0000","last_change":"2023-12-04T05:39:58.750554+0000","last_active":"2023-12-04T05:40:01.502449+0000","last_peered":"2023-12-04T05:40:01.502449+0000","last_clean":"2023-12-04T05:40:01.502449+0000","last_became_active":"2023-12-04T05:39:58.750227+0000","last_became_peered":"2023-12-04T05:39:58.750227+0000","last_unstale":"2023-12-04T05:40:01.502449+0000","last_undegraded":"2023-12-04T05:40:01.502449+0000","last_fullsized":"2023-12-04T05:40:01.502449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:13:23.997255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502276+0000","last_change":"2023-12-04T05:39:59.731582+0000","last_active":"2023-12-04T05:40:01.502276+0000","last_peered":"2023-12-04T05:40:01.502276+0000","last_clean":"2023-12-04T05:40:01.502276+0000","last_became_active":"2023-12-04T05:39:59.730822+0000","last_became_peered":"2023-12-04T05:39:59.730822+0000","last_unstale":"2023-12-04T05:40:01.502276+0000","last_undegraded":"2023-12-04T05:40:01.502276+0000","last_fullsized":"2023-12-04T05:40:01.502276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:10:55.996104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501262+0000","last_change":"2023-12-04T05:39:58.740011+0000","last_active":"2023-12-04T05:40:01.501262+0000","last_peered":"2023-12-04T05:40:01.501262+0000","last_clean":"2023-12-04T05:40:01.501262+0000","last_became_active":"2023-12-04T05:39:58.739395+0000","last_became_peered":"2023-12-04T05:39:58.739395+0000","last_unstale":"2023-12-04T05:40:01.501262+0000","last_undegraded":"2023-12-04T05:40:01.501262+0000","last_fullsized":"2023-12-04T05:40:01.501262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:17:48.107311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502182+0000","last_change":"2023-12-04T05:39:58.730186+0000","last_active":"2023-12-04T05:40:01.502182+0000","last_peered":"2023-12-04T05:40:01.502182+0000","last_clean":"2023-12-04T05:40:01.502182+0000","last_became_active":"2023-12-04T05:39:58.729411+0000","last_became_peered":"2023-12-04T05:39:58.729411+0000","last_unstale":"2023-12-04T05:40:01.502182+0000","last_undegraded":"2023-12-04T05:40:01.502182+0000","last_fullsized":"2023-12-04T05:40:01.502182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:39:05.463010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502530+0000","last_change":"2023-12-04T05:39:59.715596+0000","last_active":"2023-12-04T05:40:01.502530+0000","last_peered":"2023-12-04T05:40:01.502530+0000","last_clean":"2023-12-04T05:40:01.502530+0000","last_became_active":"2023-12-04T05:39:59.714842+0000","last_became_peered":"2023-12-04T05:39:59.714842+0000","last_unstale":"2023-12-04T05:40:01.502530+0000","last_undegraded":"2023-12-04T05:40:01.502530+0000","last_fullsized":"2023-12-04T05:40:01.502530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:51:10.874053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682760+0000","last_change":"2023-12-04T05:39:58.722113+0000","last_active":"2023-12-04T05:40:00.682760+0000","last_peered":"2023-12-04T05:40:00.682760+0000","last_clean":"2023-12-04T05:40:00.682760+0000","last_became_active":"2023-12-04T05:39:58.721346+0000","last_became_peered":"2023-12-04T05:39:58.721346+0000","last_unstale":"2023-12-04T05:40:00.682760+0000","last_undegraded":"2023-12-04T05:40:00.682760+0000","last_fullsized":"2023-12-04T05:40:00.682760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:30:11.578047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908461+0000","last_change":"2023-12-04T05:39:59.708575+0000","last_active":"2023-12-04T05:40:00.908461+0000","last_peered":"2023-12-04T05:40:00.908461+0000","last_clean":"2023-12-04T05:40:00.908461+0000","last_became_active":"2023-12-04T05:39:59.707753+0000","last_became_peered":"2023-12-04T05:39:59.707753+0000","last_unstale":"2023-12-04T05:40:00.908461+0000","last_undegraded":"2023-12-04T05:40:00.908461+0000","last_fullsized":"2023-12-04T05:40:00.908461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:47:58.349368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501888+0000","last_change":"2023-12-04T05:39:58.734739+0000","last_active":"2023-12-04T05:40:01.501888+0000","last_peered":"2023-12-04T05:40:01.501888+0000","last_clean":"2023-12-04T05:40:01.501888+0000","last_became_active":"2023-12-04T05:39:58.733897+0000","last_became_peered":"2023-12-04T05:39:58.733897+0000","last_unstale":"2023-12-04T05:40:01.501888+0000","last_undegraded":"2023-12-04T05:40:01.501888+0000","last_fullsized":"2023-12-04T05:40:01.501888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:52:15.157364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683392+0000","last_change":"2023-12-04T05:39:59.715796+0000","last_active":"2023-12-04T05:40:00.683392+0000","last_peered":"2023-12-04T05:40:00.683392+0000","last_clean":"2023-12-04T05:40:00.683392+0000","last_became_active":"2023-12-04T05:39:59.714951+0000","last_became_peered":"2023-12-04T05:39:59.714951+0000","last_unstale":"2023-12-04T05:40:00.683392+0000","last_undegraded":"2023-12-04T05:40:00.683392+0000","last_fullsized":"2023-12-04T05:40:00.683392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:31:12.475136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683603+0000","last_change":"2023-12-04T05:39:58.725387+0000","last_active":"2023-12-04T05:40:00.683603+0000","last_peered":"2023-12-04T05:40:00.683603+0000","last_clean":"2023-12-04T05:40:00.683603+0000","last_became_active":"2023-12-04T05:39:58.724672+0000","last_became_peered":"2023-12-04T05:39:58.724672+0000","last_unstale":"2023-12-04T05:40:00.683603+0000","last_undegraded":"2023-12-04T05:40:00.683603+0000","last_fullsized":"2023-12-04T05:40:00.683603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:19:43.112172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503316+0000","last_change":"2023-12-04T05:39:59.718789+0000","last_active":"2023-12-04T05:40:01.503316+0000","last_peered":"2023-12-04T05:40:01.503316+0000","last_clean":"2023-12-04T05:40:01.503316+0000","last_became_active":"2023-12-04T05:39:59.717927+0000","last_became_peered":"2023-12-04T05:39:59.717927+0000","last_unstale":"2023-12-04T05:40:01.503316+0000","last_undegraded":"2023-12-04T05:40:01.503316+0000","last_fullsized":"2023-12-04T05:40:01.503316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:43:08.771461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.909015+0000","last_change":"2023-12-04T05:39:58.731710+0000","last_active":"2023-12-04T05:40:00.909015+0000","last_peered":"2023-12-04T05:40:00.909015+0000","last_clean":"2023-12-04T05:40:00.909015+0000","last_became_active":"2023-12-04T05:39:58.731373+0000","last_became_peered":"2023-12-04T05:39:58.731373+0000","last_unstale":"2023-12-04T05:40:00.909015+0000","last_undegraded":"2023-12-04T05:40:00.909015+0000","last_fullsized":"2023-12-04T05:40:00.909015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:03:37.899195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682594+0000","last_change":"2023-12-04T05:39:59.725444+0000","last_active":"2023-12-04T05:40:00.682594+0000","last_peered":"2023-12-04T05:40:00.682594+0000","last_clean":"2023-12-04T05:40:00.682594+0000","last_became_active":"2023-12-04T05:39:59.724620+0000","last_became_peered":"2023-12-04T05:39:59.724620+0000","last_unstale":"2023-12-04T05:40:00.682594+0000","last_undegraded":"2023-12-04T05:40:00.682594+0000","last_fullsized":"2023-12-04T05:40:00.682594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:11:17.092060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501523+0000","last_change":"2023-12-04T05:39:58.730135+0000","last_active":"2023-12-04T05:40:01.501523+0000","last_peered":"2023-12-04T05:40:01.501523+0000","last_clean":"2023-12-04T05:40:01.501523+0000","last_became_active":"2023-12-04T05:39:58.729295+0000","last_became_peered":"2023-12-04T05:39:58.729295+0000","last_unstale":"2023-12-04T05:40:01.501523+0000","last_undegraded":"2023-12-04T05:40:01.501523+0000","last_fullsized":"2023-12-04T05:40:01.501523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:24:42.912063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502658+0000","last_change":"2023-12-04T05:39:59.731327+0000","last_active":"2023-12-04T05:40:01.502658+0000","last_peered":"2023-12-04T05:40:01.502658+0000","last_clean":"2023-12-04T05:40:01.502658+0000","last_became_active":"2023-12-04T05:39:59.730427+0000","last_became_peered":"2023-12-04T05:39:59.730427+0000","last_unstale":"2023-12-04T05:40:01.502658+0000","last_undegraded":"2023-12-04T05:40:01.502658+0000","last_fullsized":"2023-12-04T05:40:01.502658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:34:45.211951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502490+0000","last_change":"2023-12-04T05:39:58.711690+0000","last_active":"2023-12-04T05:40:01.502490+0000","last_peered":"2023-12-04T05:40:01.502490+0000","last_clean":"2023-12-04T05:40:01.502490+0000","last_became_active":"2023-12-04T05:39:58.711305+0000","last_became_peered":"2023-12-04T05:39:58.711305+0000","last_unstale":"2023-12-04T05:40:01.502490+0000","last_undegraded":"2023-12-04T05:40:01.502490+0000","last_fullsized":"2023-12-04T05:40:01.502490+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:33:02.159265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501048+0000","last_change":"2023-12-04T05:39:59.702650+0000","last_active":"2023-12-04T05:40:01.501048+0000","last_peered":"2023-12-04T05:40:01.501048+0000","last_clean":"2023-12-04T05:40:01.501048+0000","last_became_active":"2023-12-04T05:39:59.701827+0000","last_became_peered":"2023-12-04T05:39:59.701827+0000","last_unstale":"2023-12-04T05:40:01.501048+0000","last_undegraded":"2023-12-04T05:40:01.501048+0000","last_fullsized":"2023-12-04T05:40:01.501048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:46:45.006062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683932+0000","last_change":"2023-12-04T05:39:58.729371+0000","last_active":"2023-12-04T05:40:00.683932+0000","last_peered":"2023-12-04T05:40:00.683932+0000","last_clean":"2023-12-04T05:40:00.683932+0000","last_became_active":"2023-12-04T05:39:58.727781+0000","last_became_peered":"2023-12-04T05:39:58.727781+0000","last_unstale":"2023-12-04T05:40:00.683932+0000","last_undegraded":"2023-12-04T05:40:00.683932+0000","last_fullsized":"2023-12-04T05:40:00.683932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:37:29.707707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502646+0000","last_change":"2023-12-04T05:39:59.742080+0000","last_active":"2023-12-04T05:40:01.502646+0000","last_peered":"2023-12-04T05:40:01.502646+0000","last_clean":"2023-12-04T05:40:01.502646+0000","last_became_active":"2023-12-04T05:39:59.741440+0000","last_became_peered":"2023-12-04T05:39:59.741440+0000","last_unstale":"2023-12-04T05:40:01.502646+0000","last_undegraded":"2023-12-04T05:40:01.502646+0000","last_fullsized":"2023-12-04T05:40:01.502646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:29:21.379583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.500870+0000","last_change":"2023-12-04T05:39:58.737926+0000","last_active":"2023-12-04T05:40:01.500870+0000","last_peered":"2023-12-04T05:40:01.500870+0000","last_clean":"2023-12-04T05:40:01.500870+0000","last_became_active":"2023-12-04T05:39:58.737403+0000","last_became_peered":"2023-12-04T05:39:58.737403+0000","last_unstale":"2023-12-04T05:40:01.500870+0000","last_undegraded":"2023-12-04T05:40:01.500870+0000","last_fullsized":"2023-12-04T05:40:01.500870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:10:59.035282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501005+0000","last_change":"2023-12-04T05:39:59.710310+0000","last_active":"2023-12-04T05:40:01.501005+0000","last_peered":"2023-12-04T05:40:01.501005+0000","last_clean":"2023-12-04T05:40:01.501005+0000","last_became_active":"2023-12-04T05:39:59.709497+0000","last_became_peered":"2023-12-04T05:39:59.709497+0000","last_unstale":"2023-12-04T05:40:01.501005+0000","last_undegraded":"2023-12-04T05:40:01.501005+0000","last_fullsized":"2023-12-04T05:40:01.501005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:43:43.708311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.678550+0000","last_change":"2023-12-04T05:39:58.728046+0000","last_active":"2023-12-04T05:40:00.678550+0000","last_peered":"2023-12-04T05:40:00.678550+0000","last_clean":"2023-12-04T05:40:00.678550+0000","last_became_active":"2023-12-04T05:39:58.726747+0000","last_became_peered":"2023-12-04T05:39:58.726747+0000","last_unstale":"2023-12-04T05:40:00.678550+0000","last_undegraded":"2023-12-04T05:40:00.678550+0000","last_fullsized":"2023-12-04T05:40:00.678550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:45:11.024395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501689+0000","last_change":"2023-12-04T05:39:59.741770+0000","last_active":"2023-12-04T05:40:01.501689+0000","last_peered":"2023-12-04T05:40:01.501689+0000","last_clean":"2023-12-04T05:40:01.501689+0000","last_became_active":"2023-12-04T05:39:59.741091+0000","last_became_peered":"2023-12-04T05:39:59.741091+0000","last_unstale":"2023-12-04T05:40:01.501689+0000","last_undegraded":"2023-12-04T05:40:01.501689+0000","last_fullsized":"2023-12-04T05:40:01.501689+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:59:20.747612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682211+0000","last_change":"2023-12-04T05:39:58.712970+0000","last_active":"2023-12-04T05:40:00.682211+0000","last_peered":"2023-12-04T05:40:00.682211+0000","last_clean":"2023-12-04T05:40:00.682211+0000","last_became_active":"2023-12-04T05:39:58.711759+0000","last_became_peered":"2023-12-04T05:39:58.711759+0000","last_unstale":"2023-12-04T05:40:00.682211+0000","last_undegraded":"2023-12-04T05:40:00.682211+0000","last_fullsized":"2023-12-04T05:40:00.682211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:02:00.957563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503407+0000","last_change":"2023-12-04T05:39:59.742244+0000","last_active":"2023-12-04T05:40:01.503407+0000","last_peered":"2023-12-04T05:40:01.503407+0000","last_clean":"2023-12-04T05:40:01.503407+0000","last_became_active":"2023-12-04T05:39:59.741379+0000","last_became_peered":"2023-12-04T05:39:59.741379+0000","last_unstale":"2023-12-04T05:40:01.503407+0000","last_undegraded":"2023-12-04T05:40:01.503407+0000","last_fullsized":"2023-12-04T05:40:01.503407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:07:57.685895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.679104+0000","last_change":"2023-12-04T05:39:58.729723+0000","last_active":"2023-12-04T05:40:00.679104+0000","last_peered":"2023-12-04T05:40:00.679104+0000","last_clean":"2023-12-04T05:40:00.679104+0000","last_became_active":"2023-12-04T05:39:58.729410+0000","last_became_peered":"2023-12-04T05:39:58.729410+0000","last_unstale":"2023-12-04T05:40:00.679104+0000","last_undegraded":"2023-12-04T05:40:00.679104+0000","last_fullsized":"2023-12-04T05:40:00.679104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:59:10.799774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503914+0000","last_change":"2023-12-04T05:39:59.741718+0000","last_active":"2023-12-04T05:40:01.503914+0000","last_peered":"2023-12-04T05:40:01.503914+0000","last_clean":"2023-12-04T05:40:01.503914+0000","last_became_active":"2023-12-04T05:39:59.741101+0000","last_became_peered":"2023-12-04T05:39:59.741101+0000","last_unstale":"2023-12-04T05:40:01.503914+0000","last_undegraded":"2023-12-04T05:40:01.503914+0000","last_fullsized":"2023-12-04T05:40:01.503914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:41:03.969161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.909154+0000","last_change":"2023-12-04T05:39:58.737224+0000","last_active":"2023-12-04T05:40:00.909154+0000","last_peered":"2023-12-04T05:40:00.909154+0000","last_clean":"2023-12-04T05:40:00.909154+0000","last_became_active":"2023-12-04T05:39:58.736753+0000","last_became_peered":"2023-12-04T05:39:58.736753+0000","last_unstale":"2023-12-04T05:40:00.909154+0000","last_undegraded":"2023-12-04T05:40:00.909154+0000","last_fullsized":"2023-12-04T05:40:00.909154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:33:34.027796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683405+0000","last_change":"2023-12-04T05:39:59.726776+0000","last_active":"2023-12-04T05:40:00.683405+0000","last_peered":"2023-12-04T05:40:00.683405+0000","last_clean":"2023-12-04T05:40:00.683405+0000","last_became_active":"2023-12-04T05:39:59.725991+0000","last_became_peered":"2023-12-04T05:39:59.725991+0000","last_unstale":"2023-12-04T05:40:00.683405+0000","last_undegraded":"2023-12-04T05:40:00.683405+0000","last_fullsized":"2023-12-04T05:40:00.683405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:36:02.168898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.678822+0000","last_change":"2023-12-04T05:39:58.732942+0000","last_active":"2023-12-04T05:40:00.678822+0000","last_peered":"2023-12-04T05:40:00.678822+0000","last_clean":"2023-12-04T05:40:00.678822+0000","last_became_active":"2023-12-04T05:39:58.732401+0000","last_became_peered":"2023-12-04T05:39:58.732401+0000","last_unstale":"2023-12-04T05:40:00.678822+0000","last_undegraded":"2023-12-04T05:40:00.678822+0000","last_fullsized":"2023-12-04T05:40:00.678822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:23:13.676061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503215+0000","last_change":"2023-12-04T05:39:59.730768+0000","last_active":"2023-12-04T05:40:01.503215+0000","last_peered":"2023-12-04T05:40:01.503215+0000","last_clean":"2023-12-04T05:40:01.503215+0000","last_became_active":"2023-12-04T05:39:59.730207+0000","last_became_peered":"2023-12-04T05:39:59.730207+0000","last_unstale":"2023-12-04T05:40:01.503215+0000","last_undegraded":"2023-12-04T05:40:01.503215+0000","last_fullsized":"2023-12-04T05:40:01.503215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:25:19.473188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T05:39:59.705581+0000","last_change":"2023-12-04T05:39:49.314208+0000","last_active":"2023-12-04T05:39:59.705581+0000","last_peered":"2023-12-04T05:39:59.705581+0000","last_clean":"2023-12-04T05:39:59.705581+0000","last_became_active":"2023-12-04T05:39:49.312262+0000","last_became_peered":"2023-12-04T05:39:49.312262+0000","last_unstale":"2023-12-04T05:39:59.705581+0000","last_undegraded":"2023-12-04T05:39:59.705581+0000","last_fullsized":"2023-12-04T05:39:59.705581+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:48.242612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:48.242612+0000","last_clean_scrub_stamp":"2023-12-04T05:39:48.242612+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:19:26.511345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682745+0000","last_change":"2023-12-04T05:39:58.725538+0000","last_active":"2023-12-04T05:40:00.682745+0000","last_peered":"2023-12-04T05:40:00.682745+0000","last_clean":"2023-12-04T05:40:00.682745+0000","last_became_active":"2023-12-04T05:39:58.724839+0000","last_became_peered":"2023-12-04T05:39:58.724839+0000","last_unstale":"2023-12-04T05:40:00.682745+0000","last_undegraded":"2023-12-04T05:40:00.682745+0000","last_fullsized":"2023-12-04T05:40:00.682745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:34:48.521117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501800+0000","last_change":"2023-12-04T05:39:59.719475+0000","last_active":"2023-12-04T05:40:01.501800+0000","last_peered":"2023-12-04T05:40:01.501800+0000","last_clean":"2023-12-04T05:40:01.501800+0000","last_became_active":"2023-12-04T05:39:59.718727+0000","last_became_peered":"2023-12-04T05:39:59.718727+0000","last_unstale":"2023-12-04T05:40:01.501800+0000","last_undegraded":"2023-12-04T05:40:01.501800+0000","last_fullsized":"2023-12-04T05:40:01.501800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:53:35.787086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501396+0000","last_change":"2023-12-04T05:39:58.751135+0000","last_active":"2023-12-04T05:40:01.501396+0000","last_peered":"2023-12-04T05:40:01.501396+0000","last_clean":"2023-12-04T05:40:01.501396+0000","last_became_active":"2023-12-04T05:39:58.750144+0000","last_became_peered":"2023-12-04T05:39:58.750144+0000","last_unstale":"2023-12-04T05:40:01.501396+0000","last_undegraded":"2023-12-04T05:40:01.501396+0000","last_fullsized":"2023-12-04T05:40:01.501396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:31:32.599659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.500789+0000","last_change":"2023-12-04T05:39:59.729585+0000","last_active":"2023-12-04T05:40:01.500789+0000","last_peered":"2023-12-04T05:40:01.500789+0000","last_clean":"2023-12-04T05:40:01.500789+0000","last_became_active":"2023-12-04T05:39:59.728381+0000","last_became_peered":"2023-12-04T05:39:59.728381+0000","last_unstale":"2023-12-04T05:40:01.500789+0000","last_undegraded":"2023-12-04T05:40:01.500789+0000","last_fullsized":"2023-12-04T05:40:01.500789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:35:11.920755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.678836+0000","last_change":"2023-12-04T05:39:58.731429+0000","last_active":"2023-12-04T05:40:00.678836+0000","last_peered":"2023-12-04T05:40:00.678836+0000","last_clean":"2023-12-04T05:40:00.678836+0000","last_became_active":"2023-12-04T05:39:58.731049+0000","last_became_peered":"2023-12-04T05:39:58.731049+0000","last_unstale":"2023-12-04T05:40:00.678836+0000","last_undegraded":"2023-12-04T05:40:00.678836+0000","last_fullsized":"2023-12-04T05:40:00.678836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:27:16.863460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908531+0000","last_change":"2023-12-04T05:39:59.701324+0000","last_active":"2023-12-04T05:40:00.908531+0000","last_peered":"2023-12-04T05:40:00.908531+0000","last_clean":"2023-12-04T05:40:00.908531+0000","last_became_active":"2023-12-04T05:39:59.700586+0000","last_became_peered":"2023-12-04T05:39:59.700586+0000","last_unstale":"2023-12-04T05:40:00.908531+0000","last_undegraded":"2023-12-04T05:40:00.908531+0000","last_fullsized":"2023-12-04T05:40:00.908531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:37:17.301688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.684152+0000","last_change":"2023-12-04T05:39:58.729635+0000","last_active":"2023-12-04T05:40:00.684152+0000","last_peered":"2023-12-04T05:40:00.684152+0000","last_clean":"2023-12-04T05:40:00.684152+0000","last_became_active":"2023-12-04T05:39:58.728927+0000","last_became_peered":"2023-12-04T05:39:58.728927+0000","last_unstale":"2023-12-04T05:40:00.684152+0000","last_undegraded":"2023-12-04T05:40:00.684152+0000","last_fullsized":"2023-12-04T05:40:00.684152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:58:46.851065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681790+0000","last_change":"2023-12-04T05:39:59.702840+0000","last_active":"2023-12-04T05:40:00.681790+0000","last_peered":"2023-12-04T05:40:00.681790+0000","last_clean":"2023-12-04T05:40:00.681790+0000","last_became_active":"2023-12-04T05:39:59.701652+0000","last_became_peered":"2023-12-04T05:39:59.701652+0000","last_unstale":"2023-12-04T05:40:00.681790+0000","last_undegraded":"2023-12-04T05:40:00.681790+0000","last_fullsized":"2023-12-04T05:40:00.681790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:44:14.733182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908769+0000","last_change":"2023-12-04T05:39:58.721834+0000","last_active":"2023-12-04T05:40:00.908769+0000","last_peered":"2023-12-04T05:40:00.908769+0000","last_clean":"2023-12-04T05:40:00.908769+0000","last_became_active":"2023-12-04T05:39:58.721451+0000","last_became_peered":"2023-12-04T05:39:58.721451+0000","last_unstale":"2023-12-04T05:40:00.908769+0000","last_undegraded":"2023-12-04T05:40:00.908769+0000","last_fullsized":"2023-12-04T05:40:00.908769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:15:52.065853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.684129+0000","last_change":"2023-12-04T05:39:59.731053+0000","last_active":"2023-12-04T05:40:00.684129+0000","last_peered":"2023-12-04T05:40:00.684129+0000","last_clean":"2023-12-04T05:40:00.684129+0000","last_became_active":"2023-12-04T05:39:59.729732+0000","last_became_peered":"2023-12-04T05:39:59.729732+0000","last_unstale":"2023-12-04T05:40:00.684129+0000","last_undegraded":"2023-12-04T05:40:00.684129+0000","last_fullsized":"2023-12-04T05:40:00.684129+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:57:38.581311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503452+0000","last_change":"2023-12-04T05:39:58.751126+0000","last_active":"2023-12-04T05:40:01.503452+0000","last_peered":"2023-12-04T05:40:01.503452+0000","last_clean":"2023-12-04T05:40:01.503452+0000","last_became_active":"2023-12-04T05:39:58.750496+0000","last_became_peered":"2023-12-04T05:39:58.750496+0000","last_unstale":"2023-12-04T05:40:01.503452+0000","last_undegraded":"2023-12-04T05:40:01.503452+0000","last_fullsized":"2023-12-04T05:40:01.503452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:09:22.333993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908579+0000","last_change":"2023-12-04T05:39:59.700032+0000","last_active":"2023-12-04T05:40:00.908579+0000","last_peered":"2023-12-04T05:40:00.908579+0000","last_clean":"2023-12-04T05:40:00.908579+0000","last_became_active":"2023-12-04T05:39:59.699488+0000","last_became_peered":"2023-12-04T05:39:59.699488+0000","last_unstale":"2023-12-04T05:40:00.908579+0000","last_undegraded":"2023-12-04T05:40:00.908579+0000","last_fullsized":"2023-12-04T05:40:00.908579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:59:57.153486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.909014+0000","last_change":"2023-12-04T05:39:58.723126+0000","last_active":"2023-12-04T05:40:00.909014+0000","last_peered":"2023-12-04T05:40:00.909014+0000","last_clean":"2023-12-04T05:40:00.909014+0000","last_became_active":"2023-12-04T05:39:58.722766+0000","last_became_peered":"2023-12-04T05:39:58.722766+0000","last_unstale":"2023-12-04T05:40:00.909014+0000","last_undegraded":"2023-12-04T05:40:00.909014+0000","last_fullsized":"2023-12-04T05:40:00.909014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:28:47.171345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502918+0000","last_change":"2023-12-04T05:39:59.724653+0000","last_active":"2023-12-04T05:40:01.502918+0000","last_peered":"2023-12-04T05:40:01.502918+0000","last_clean":"2023-12-04T05:40:01.502918+0000","last_became_active":"2023-12-04T05:39:59.723945+0000","last_became_peered":"2023-12-04T05:39:59.723945+0000","last_unstale":"2023-12-04T05:40:01.502918+0000","last_undegraded":"2023-12-04T05:40:01.502918+0000","last_fullsized":"2023-12-04T05:40:01.502918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:13:56.812140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501945+0000","last_change":"2023-12-04T05:39:58.746966+0000","last_active":"2023-12-04T05:40:01.501945+0000","last_peered":"2023-12-04T05:40:01.501945+0000","last_clean":"2023-12-04T05:40:01.501945+0000","last_became_active":"2023-12-04T05:39:58.746261+0000","last_became_peered":"2023-12-04T05:39:58.746261+0000","last_unstale":"2023-12-04T05:40:01.501945+0000","last_undegraded":"2023-12-04T05:40:01.501945+0000","last_fullsized":"2023-12-04T05:40:01.501945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:57:46.899291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503381+0000","last_change":"2023-12-04T05:39:59.713259+0000","last_active":"2023-12-04T05:40:01.503381+0000","last_peered":"2023-12-04T05:40:01.503381+0000","last_clean":"2023-12-04T05:40:01.503381+0000","last_became_active":"2023-12-04T05:39:59.712523+0000","last_became_peered":"2023-12-04T05:39:59.712523+0000","last_unstale":"2023-12-04T05:40:01.503381+0000","last_undegraded":"2023-12-04T05:40:01.503381+0000","last_fullsized":"2023-12-04T05:40:01.503381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:41:10.001755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501174+0000","last_change":"2023-12-04T05:39:58.750282+0000","last_active":"2023-12-04T05:40:01.501174+0000","last_peered":"2023-12-04T05:40:01.501174+0000","last_clean":"2023-12-04T05:40:01.501174+0000","last_became_active":"2023-12-04T05:39:58.749651+0000","last_became_peered":"2023-12-04T05:39:58.749651+0000","last_unstale":"2023-12-04T05:40:01.501174+0000","last_undegraded":"2023-12-04T05:40:01.501174+0000","last_fullsized":"2023-12-04T05:40:01.501174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:40:55.086649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681687+0000","last_change":"2023-12-04T05:39:59.704667+0000","last_active":"2023-12-04T05:40:00.681687+0000","last_peered":"2023-12-04T05:40:00.681687+0000","last_clean":"2023-12-04T05:40:00.681687+0000","last_became_active":"2023-12-04T05:39:59.703943+0000","last_became_peered":"2023-12-04T05:39:59.703943+0000","last_unstale":"2023-12-04T05:40:00.681687+0000","last_undegraded":"2023-12-04T05:40:00.681687+0000","last_fullsized":"2023-12-04T05:40:00.681687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:31:53.890532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908698+0000","last_change":"2023-12-04T05:39:58.745277+0000","last_active":"2023-12-04T05:40:00.908698+0000","last_peered":"2023-12-04T05:40:00.908698+0000","last_clean":"2023-12-04T05:40:00.908698+0000","last_became_active":"2023-12-04T05:39:58.744842+0000","last_became_peered":"2023-12-04T05:39:58.744842+0000","last_unstale":"2023-12-04T05:40:00.908698+0000","last_undegraded":"2023-12-04T05:40:00.908698+0000","last_fullsized":"2023-12-04T05:40:00.908698+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:50:21.084855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502964+0000","last_change":"2023-12-04T05:39:59.714217+0000","last_active":"2023-12-04T05:40:01.502964+0000","last_peered":"2023-12-04T05:40:01.502964+0000","last_clean":"2023-12-04T05:40:01.502964+0000","last_became_active":"2023-12-04T05:39:59.713438+0000","last_became_peered":"2023-12-04T05:39:59.713438+0000","last_unstale":"2023-12-04T05:40:01.502964+0000","last_undegraded":"2023-12-04T05:40:01.502964+0000","last_fullsized":"2023-12-04T05:40:01.502964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:37:30.790544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908612+0000","last_change":"2023-12-04T05:39:58.722048+0000","last_active":"2023-12-04T05:40:00.908612+0000","last_peered":"2023-12-04T05:40:00.908612+0000","last_clean":"2023-12-04T05:40:00.908612+0000","last_became_active":"2023-12-04T05:39:58.721782+0000","last_became_peered":"2023-12-04T05:39:58.721782+0000","last_unstale":"2023-12-04T05:40:00.908612+0000","last_undegraded":"2023-12-04T05:40:00.908612+0000","last_fullsized":"2023-12-04T05:40:00.908612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:38:16.023373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502222+0000","last_change":"2023-12-04T05:39:59.715313+0000","last_active":"2023-12-04T05:40:01.502222+0000","last_peered":"2023-12-04T05:40:01.502222+0000","last_clean":"2023-12-04T05:40:01.502222+0000","last_became_active":"2023-12-04T05:39:59.714430+0000","last_became_peered":"2023-12-04T05:39:59.714430+0000","last_unstale":"2023-12-04T05:40:01.502222+0000","last_undegraded":"2023-12-04T05:40:01.502222+0000","last_fullsized":"2023-12-04T05:40:01.502222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:10:01.533077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502795+0000","last_change":"2023-12-04T05:39:58.750921+0000","last_active":"2023-12-04T05:40:01.502795+0000","last_peered":"2023-12-04T05:40:01.502795+0000","last_clean":"2023-12-04T05:40:01.502795+0000","last_became_active":"2023-12-04T05:39:58.750288+0000","last_became_peered":"2023-12-04T05:39:58.750288+0000","last_unstale":"2023-12-04T05:40:01.502795+0000","last_undegraded":"2023-12-04T05:40:01.502795+0000","last_fullsized":"2023-12-04T05:40:01.502795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:04:26.373696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501279+0000","last_change":"2023-12-04T05:39:59.729260+0000","last_active":"2023-12-04T05:40:01.501279+0000","last_peered":"2023-12-04T05:40:01.501279+0000","last_clean":"2023-12-04T05:40:01.501279+0000","last_became_active":"2023-12-04T05:39:59.728639+0000","last_became_peered":"2023-12-04T05:39:59.728639+0000","last_unstale":"2023-12-04T05:40:01.501279+0000","last_undegraded":"2023-12-04T05:40:01.501279+0000","last_fullsized":"2023-12-04T05:40:01.501279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:21:38.039584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683814+0000","last_change":"2023-12-04T05:39:59.731652+0000","last_active":"2023-12-04T05:40:00.683814+0000","last_peered":"2023-12-04T05:40:00.683814+0000","last_clean":"2023-12-04T05:40:00.683814+0000","last_became_active":"2023-12-04T05:39:59.730714+0000","last_became_peered":"2023-12-04T05:39:59.730714+0000","last_unstale":"2023-12-04T05:40:00.683814+0000","last_undegraded":"2023-12-04T05:40:00.683814+0000","last_fullsized":"2023-12-04T05:40:00.683814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:02:27.013221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502732+0000","last_change":"2023-12-04T05:39:58.751530+0000","last_active":"2023-12-04T05:40:01.502732+0000","last_peered":"2023-12-04T05:40:01.502732+0000","last_clean":"2023-12-04T05:40:01.502732+0000","last_became_active":"2023-12-04T05:39:58.750857+0000","last_became_peered":"2023-12-04T05:39:58.750857+0000","last_unstale":"2023-12-04T05:40:01.502732+0000","last_undegraded":"2023-12-04T05:40:01.502732+0000","last_fullsized":"2023-12-04T05:40:01.502732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:43:36.369813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683195+0000","last_change":"2023-12-04T05:39:59.730535+0000","last_active":"2023-12-04T05:40:00.683195+0000","last_peered":"2023-12-04T05:40:00.683195+0000","last_clean":"2023-12-04T05:40:00.683195+0000","last_became_active":"2023-12-04T05:39:59.729717+0000","last_became_peered":"2023-12-04T05:39:59.729717+0000","last_unstale":"2023-12-04T05:40:00.683195+0000","last_undegraded":"2023-12-04T05:40:00.683195+0000","last_fullsized":"2023-12-04T05:40:00.683195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:31:46.120414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502974+0000","last_change":"2023-12-04T05:39:58.752193+0000","last_active":"2023-12-04T05:40:01.502974+0000","last_peered":"2023-12-04T05:40:01.502974+0000","last_clean":"2023-12-04T05:40:01.502974+0000","last_became_active":"2023-12-04T05:39:58.751625+0000","last_became_peered":"2023-12-04T05:39:58.751625+0000","last_unstale":"2023-12-04T05:40:01.502974+0000","last_undegraded":"2023-12-04T05:40:01.502974+0000","last_fullsized":"2023-12-04T05:40:01.502974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:05:05.734494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501167+0000","last_change":"2023-12-04T05:39:59.720197+0000","last_active":"2023-12-04T05:40:01.501167+0000","last_peered":"2023-12-04T05:40:01.501167+0000","last_clean":"2023-12-04T05:40:01.501167+0000","last_became_active":"2023-12-04T05:39:59.718736+0000","last_became_peered":"2023-12-04T05:39:59.718736+0000","last_unstale":"2023-12-04T05:40:01.501167+0000","last_undegraded":"2023-12-04T05:40:01.501167+0000","last_fullsized":"2023-12-04T05:40:01.501167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:41:13.184198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503280+0000","last_change":"2023-12-04T05:39:58.726906+0000","last_active":"2023-12-04T05:40:01.503280+0000","last_peered":"2023-12-04T05:40:01.503280+0000","last_clean":"2023-12-04T05:40:01.503280+0000","last_became_active":"2023-12-04T05:39:58.726415+0000","last_became_peered":"2023-12-04T05:39:58.726415+0000","last_unstale":"2023-12-04T05:40:01.503280+0000","last_undegraded":"2023-12-04T05:40:01.503280+0000","last_fullsized":"2023-12-04T05:40:01.503280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:58:53.715291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503085+0000","last_change":"2023-12-04T05:39:59.711861+0000","last_active":"2023-12-04T05:40:01.503085+0000","last_peered":"2023-12-04T05:40:01.503085+0000","last_clean":"2023-12-04T05:40:01.503085+0000","last_became_active":"2023-12-04T05:39:59.710933+0000","last_became_peered":"2023-12-04T05:39:59.710933+0000","last_unstale":"2023-12-04T05:40:01.503085+0000","last_undegraded":"2023-12-04T05:40:01.503085+0000","last_fullsized":"2023-12-04T05:40:01.503085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:40:46.361768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501520+0000","last_change":"2023-12-04T05:39:58.740704+0000","last_active":"2023-12-04T05:40:01.501520+0000","last_peered":"2023-12-04T05:40:01.501520+0000","last_clean":"2023-12-04T05:40:01.501520+0000","last_became_active":"2023-12-04T05:39:58.740440+0000","last_became_peered":"2023-12-04T05:39:58.740440+0000","last_unstale":"2023-12-04T05:40:01.501520+0000","last_undegraded":"2023-12-04T05:40:01.501520+0000","last_fullsized":"2023-12-04T05:40:01.501520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:20:18.197917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":798720,"data_stored":659649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0589999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0940000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3879999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31380,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340460,"statfs":{"total":96636764160,"available":96604631040,"internally_reserved":0,"allocated":204800,"data_stored":69281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.26}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.163}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0169999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.375}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2270000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":69281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.111}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.052}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.476}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0580000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0960000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1200000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3080000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4399999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":798720,"data_stored":659649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-04T05:40:06.140 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-04T05:40:06.140 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-04T05:40:06.140 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-04T05:40:06.140 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-04T05:40:06.141 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-04T05:40:06.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.295+0000 7f4e17c56640 1 Processor -- start 2023-12-04T05:40:06.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.295+0000 7f4e17c56640 1 -- start start 2023-12-04T05:40:06.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e17c56640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4e10072110 0x7f4e10158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:06.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e17c56640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e10158da0 con 0x7f4e100c56f0 2023-12-04T05:40:06.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e17c56640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e101583d0 con 0x7f4e100724f0 2023-12-04T05:40:06.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e17c56640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e100c6170 con 0x7f4e10072110 2023-12-04T05:40:06.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e161f3640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4e100c56f0 0x7f4e10140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60332/0 (socket says 172.21.15.107:60332) 2023-12-04T05:40:06.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e161f3640 1 -- 172.21.15.107:0/3081973231 learned_addr learned my addr 172.21.15.107:0/3081973231 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:06.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e159f2640 1 --2- 172.21.15.107:0/3081973231 >> v2:172.21.15.195:3300/0 conn(0x7f4e10072110 0x7f4e10158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:06.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e149f0640 1 -- 172.21.15.107:0/3081973231 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123669220 0 0) 0x7f4e080014a0 con 0x7f4e100c56f0 2023-12-04T05:40:06.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e149f0640 1 -- 172.21.15.107:0/3081973231 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dfc003530 con 0x7f4e100c56f0 2023-12-04T05:40:06.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e149f0640 1 -- 172.21.15.107:0/3081973231 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 552279376 0 0) 0x7f4dfc003530 con 0x7f4e100c56f0 2023-12-04T05:40:06.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e149f0640 1 -- 172.21.15.107:0/3081973231 >> v2:172.21.15.195:3300/0 conn(0x7f4e10072110 msgr2=0x7f4e10158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:06.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e149f0640 1 --2- 172.21.15.107:0/3081973231 >> v2:172.21.15.195:3300/0 conn(0x7f4e10072110 0x7f4e10158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e149f0640 1 -- 172.21.15.107:0/3081973231 >> v1:172.21.15.195:6789/0 conn(0x7f4e100724f0 legacy=0x7f4e100c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:06.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e149f0640 1 -- 172.21.15.107:0/3081973231 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e10157da0 con 0x7f4e100c56f0 2023-12-04T05:40:06.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e149f0640 1 -- 172.21.15.107:0/3081973231 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f4e08001b90 con 0x7f4e100c56f0 2023-12-04T05:40:06.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e149f0640 1 -- 172.21.15.107:0/3081973231 >> v1:172.21.15.107:6789/0 conn(0x7f4e100c56f0 legacy=0x7f4e10140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:06.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e149f0640 1 --2- 172.21.15.107:0/3081973231 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4dfc003980 0x7f4dfc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:06.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e149f0640 1 --2- 172.21.15.107:0/3081973231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dfc0047d0 0x7f4dfc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:06.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e149f0640 1 --2- 172.21.15.107:0/3081973231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4dfc005190 0x7f4dfc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:06.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e149f0640 1 -- 172.21.15.107:0/3081973231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e100c6170 con 0x7f4dfc0047d0 2023-12-04T05:40:06.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e149f0640 1 -- 172.21.15.107:0/3081973231 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc00c040 con 0x7f4dfc005190 2023-12-04T05:40:06.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e149f0640 1 -- 172.21.15.107:0/3081973231 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc00c200 con 0x7f4dfc003980 2023-12-04T05:40:06.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e151f1640 1 --2- 172.21.15.107:0/3081973231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dfc0047d0 0x7f4dfc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:06.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e159f2640 1 --2- 172.21.15.107:0/3081973231 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4dfc003980 0x7f4dfc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:06.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e161f3640 1 --2- 172.21.15.107:0/3081973231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4dfc005190 0x7f4dfc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:06.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e151f1640 1 -- 172.21.15.107:0/3081973231 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4dfc003980 msgr2=0x7f4dfc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:06.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e151f1640 1 --2- 172.21.15.107:0/3081973231 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4dfc003980 0x7f4dfc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e151f1640 1 -- 172.21.15.107:0/3081973231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4dfc005190 msgr2=0x7f4dfc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:06.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e151f1640 1 --2- 172.21.15.107:0/3081973231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4dfc005190 0x7f4dfc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e151f1640 1 -- 172.21.15.107:0/3081973231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dfc003530 con 0x7f4dfc0047d0 2023-12-04T05:40:06.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e161f3640 1 --2- 172.21.15.107:0/3081973231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4dfc005190 0x7f4dfc00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:06.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.299+0000 7f4e161f3640 1 -- 172.21.15.107:0/3081973231 reap_dead start 2023-12-04T05:40:06.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e151f1640 1 --2- 172.21.15.107:0/3081973231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dfc0047d0 0x7f4dfc004c50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4e00001750 tx=0x7f4e0001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f7dc1baf4b6010 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:06.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e149f0640 1 -- 172.21.15.107:0/3081973231 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e000052d0 con 0x7f4dfc0047d0 2023-12-04T05:40:06.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e149f0640 1 -- 172.21.15.107:0/3081973231 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e000a7050 con 0x7f4dfc0047d0 2023-12-04T05:40:06.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e149f0640 1 -- 172.21.15.107:0/3081973231 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e00003600 con 0x7f4dfc0047d0 2023-12-04T05:40:06.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e17c56640 1 -- 172.21.15.107:0/3081973231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dfc0047d0 msgr2=0x7f4dfc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:06.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e17c56640 1 --2- 172.21.15.107:0/3081973231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dfc0047d0 0x7f4dfc004c50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4e00001750 tx=0x7f4e0001c3a0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e17c56640 1 -- 172.21.15.107:0/3081973231 shutdown_connections 2023-12-04T05:40:06.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e17c56640 1 --2- 172.21.15.107:0/3081973231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dfc0047d0 0x7f4dfc004c50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e17c56640 1 -- 172.21.15.107:0/3081973231 >> 172.21.15.107:0/3081973231 conn(0x7f4e100c7ff0 msgr2=0x7f4e100c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:06.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e17c56640 1 -- 172.21.15.107:0/3081973231 shutdown_connections 2023-12-04T05:40:06.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e17c56640 1 -- 172.21.15.107:0/3081973231 wait complete. 2023-12-04T05:40:06.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e17c56640 1 Processor -- start 2023-12-04T05:40:06.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e17c56640 1 -- start start 2023-12-04T05:40:06.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e17c56640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4dfc0047d0 0x7f4e1015bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:06.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e17c56640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e10153210 0x7f4e1015a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:06.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e17c56640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4e10149350 0x7f4e101497d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:06.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e17c56640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1014ac10 con 0x7f4e10153210 2023-12-04T05:40:06.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e17c56640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1015c040 con 0x7f4dfc0047d0 2023-12-04T05:40:06.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e17c56640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1014aeb0 con 0x7f4e10149350 2023-12-04T05:40:06.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e151f1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e10153210 0x7f4e1015a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:06.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e151f1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e10153210 0x7f4e1015a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33922/0 (socket says 172.21.15.107:33922) 2023-12-04T05:40:06.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e151f1640 1 -- 172.21.15.107:0/1348078700 learned_addr learned my addr 172.21.15.107:0/1348078700 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:06.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e159f2640 1 --2- 172.21.15.107:0/1348078700 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4dfc0047d0 0x7f4e1015bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:06.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e161f3640 1 --2- 172.21.15.107:0/1348078700 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4e10149350 0x7f4e101497d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:06.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e151f1640 1 -- 172.21.15.107:0/1348078700 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4e10149350 msgr2=0x7f4e101497d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:06.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e151f1640 1 --2- 172.21.15.107:0/1348078700 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4e10149350 0x7f4e101497d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e151f1640 1 -- 172.21.15.107:0/1348078700 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4dfc0047d0 msgr2=0x7f4e1015bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:06.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e151f1640 1 --2- 172.21.15.107:0/1348078700 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4dfc0047d0 0x7f4e1015bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.303+0000 7f4e151f1640 1 -- 172.21.15.107:0/1348078700 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e10159b10 con 0x7f4e10153210 2023-12-04T05:40:06.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.307+0000 7f4e151f1640 1 --2- 172.21.15.107:0/1348078700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e10153210 0x7f4e1015a5f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4e0001c1d0 tx=0x7f4dfc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:06.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.307+0000 7f4e067fc640 1 -- 172.21.15.107:0/1348078700 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e00007440 con 0x7f4e10153210 2023-12-04T05:40:06.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.307+0000 7f4e067fc640 1 -- 172.21.15.107:0/1348078700 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e000075a0 con 0x7f4e10153210 2023-12-04T05:40:06.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.307+0000 7f4e067fc640 1 -- 172.21.15.107:0/1348078700 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e00006a20 con 0x7f4e10153210 2023-12-04T05:40:06.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.307+0000 7f4e17c56640 1 -- 172.21.15.107:0/1348078700 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e10159df0 con 0x7f4e10153210 2023-12-04T05:40:06.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.307+0000 7f4e17c56640 1 -- 172.21.15.107:0/1348078700 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e10149d10 con 0x7f4e10153210 2023-12-04T05:40:06.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.307+0000 7f4e17c56640 1 -- 172.21.15.107:0/1348078700 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e10159950 con 0x7f4e10153210 2023-12-04T05:40:06.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.307+0000 7f4e067fc640 1 -- 172.21.15.107:0/1348078700 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4e00006b80 con 0x7f4e10153210 2023-12-04T05:40:06.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.307+0000 7f4e067fc640 1 --2- 172.21.15.107:0/1348078700 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4df40668d0 0x7f4df4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:06.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.307+0000 7f4e067fc640 1 -- 172.21.15.107:0/1348078700 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4e0011e630 con 0x7f4e10153210 2023-12-04T05:40:06.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.307+0000 7f4e159f2640 1 --2- 172.21.15.107:0/1348078700 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4df40668d0 0x7f4df4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:06.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.307+0000 7f4e159f2640 1 --2- 172.21.15.107:0/1348078700 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4df40668d0 0x7f4df4068d90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c009ab0 tx=0x7f4e0c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:06.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.311+0000 7f4e067fc640 1 -- 172.21.15.107:0/1348078700 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4e000eb870 con 0x7f4e10153210 2023-12-04T05:40:06.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.503+0000 7f4e17c56640 1 -- 172.21.15.107:0/1348078700 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f4e10067870 con 0x7f4e10153210 2023-12-04T05:40:06.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.507+0000 7f4e067fc640 1 -- 172.21.15.107:0/1348078700 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+164976 (secure 0 0 0) 0x7f4e000aae80 con 0x7f4e10153210 2023-12-04T05:40:06.510 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.507+0000 7f4e17c56640 1 -- 172.21.15.107:0/1348078700 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4df40668d0 msgr2=0x7f4df4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.507+0000 7f4e17c56640 1 --2- 172.21.15.107:0/1348078700 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4df40668d0 0x7f4df4068d90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c009ab0 tx=0x7f4e0c008040 comp rx=0 tx=0).stop 2023-12-04T05:40:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.507+0000 7f4e17c56640 1 -- 172.21.15.107:0/1348078700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e10153210 msgr2=0x7f4e1015a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.507+0000 7f4e17c56640 1 --2- 172.21.15.107:0/1348078700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e10153210 0x7f4e1015a5f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4e0001c1d0 tx=0x7f4dfc005800 comp rx=0 tx=0).stop 2023-12-04T05:40:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.511+0000 7f4e17c56640 1 -- 172.21.15.107:0/1348078700 shutdown_connections 2023-12-04T05:40:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.511+0000 7f4e17c56640 1 --2- 172.21.15.107:0/1348078700 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4df40668d0 0x7f4df4068d90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.511+0000 7f4e17c56640 1 --2- 172.21.15.107:0/1348078700 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4e10149350 0x7f4e101497d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.511+0000 7f4e17c56640 1 --2- 172.21.15.107:0/1348078700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e10153210 0x7f4e1015a5f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.511+0000 7f4e17c56640 1 --2- 172.21.15.107:0/1348078700 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4dfc0047d0 0x7f4e1015bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.511+0000 7f4e17c56640 1 -- 172.21.15.107:0/1348078700 >> 172.21.15.107:0/1348078700 conn(0x7f4e100c7ff0 msgr2=0x7f4e101572c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.511+0000 7f4e17c56640 1 -- 172.21.15.107:0/1348078700 shutdown_connections 2023-12-04T05:40:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.511+0000 7f4e17c56640 1 -- 172.21.15.107:0/1348078700 wait complete. 2023-12-04T05:40:06.529 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":5,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2693048026},{"type":"v1","addr":"172.21.15.107:6833","nonce":2693048026}]},"active_addr":"172.21.15.107:6833/2693048026","active_change":"2023-12-04T05:39:39.477445+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4105,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":2015486656}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":4257849324}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":3937995333}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":719167948}]}]} 2023-12-04T05:40:06.533 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-04T05:40:06.533 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-04T05:40:06.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T05:40:06.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.651+0000 7fbdd96d1640 1 Processor -- start 2023-12-04T05:40:06.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.651+0000 7fbdd96d1640 1 -- start start 2023-12-04T05:40:06.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.655+0000 7fbdd96d1640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbdd4157180 0x7fbdd40c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:06.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.655+0000 7fbdd96d1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbdd40c4a80 0x7fbdd40c7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:06.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.655+0000 7fbdd96d1640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd406e660 con 0x7fbdd40c46a0 2023-12-04T05:40:06.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.655+0000 7fbdd96d1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd4156490 con 0x7fbdd40c4a80 2023-12-04T05:40:06.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.655+0000 7fbdd96d1640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd4156cf0 con 0x7fbdd4157180 2023-12-04T05:40:06.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.655+0000 7fbdd37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbdd40c4a80 0x7fbdd40c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:06.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.655+0000 7fbdd37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbdd40c4a80 0x7fbdd40c7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33934/0 (socket says 172.21.15.107:33934) 2023-12-04T05:40:06.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.655+0000 7fbdd37fe640 1 -- 172.21.15.107:0/437530316 learned_addr learned my addr 172.21.15.107:0/437530316 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:06.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.655+0000 7fbdcbfff640 1 --2- 172.21.15.107:0/437530316 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbdd4157180 0x7fbdd40c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:06.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.655+0000 7fbdd37fe640 1 -- 172.21.15.107:0/437530316 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbdd4157180 msgr2=0x7fbdd40c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:06.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.655+0000 7fbdd37fe640 1 --2- 172.21.15.107:0/437530316 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbdd4157180 0x7fbdd40c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.655+0000 7fbdd37fe640 1 -- 172.21.15.107:0/437530316 >> v1:172.21.15.195:6789/0 conn(0x7fbdd40c46a0 legacy=0x7fbdd40c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:06.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.655+0000 7fbdd37fe640 1 -- 172.21.15.107:0/437530316 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd406e0a0 con 0x7fbdd40c4a80 2023-12-04T05:40:06.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.655+0000 7fbdcbfff640 1 --2- 172.21.15.107:0/437530316 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbdd4157180 0x7fbdd40c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:06.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.655+0000 7fbdd37fe640 1 --2- 172.21.15.107:0/437530316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd40c4a80 0x7fbdd40c7210 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc009e50 tx=0x7fbdcc01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d32fcc4481766f8 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:06.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.655+0000 7fbdd27fc640 1 -- 172.21.15.107:0/437530316 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc0a57d0 con 0x7fbdd40c4a80 2023-12-04T05:40:06.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.655+0000 7fbdd27fc640 1 -- 172.21.15.107:0/437530316 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdcc0a5930 con 0x7fbdd40c4a80 2023-12-04T05:40:06.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.655+0000 7fbdd27fc640 1 -- 172.21.15.107:0/437530316 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc0a5cc0 con 0x7fbdd40c4a80 2023-12-04T05:40:06.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd96d1640 1 -- 172.21.15.107:0/437530316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd40c4a80 msgr2=0x7fbdd40c7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:06.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd96d1640 1 --2- 172.21.15.107:0/437530316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd40c4a80 0x7fbdd40c7210 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc009e50 tx=0x7fbdcc01ca10 comp rx=0 tx=0).stop 2023-12-04T05:40:06.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd96d1640 1 -- 172.21.15.107:0/437530316 shutdown_connections 2023-12-04T05:40:06.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd96d1640 1 --2- 172.21.15.107:0/437530316 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbdd4157180 0x7fbdd40c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd96d1640 1 --2- 172.21.15.107:0/437530316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd40c4a80 0x7fbdd40c7210 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd96d1640 1 -- 172.21.15.107:0/437530316 >> 172.21.15.107:0/437530316 conn(0x7fbdd40c7ff0 msgr2=0x7fbdd40c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:06.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd96d1640 1 -- 172.21.15.107:0/437530316 shutdown_connections 2023-12-04T05:40:06.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd96d1640 1 -- 172.21.15.107:0/437530316 wait complete. 2023-12-04T05:40:06.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd96d1640 1 Processor -- start 2023-12-04T05:40:06.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd96d1640 1 -- start start 2023-12-04T05:40:06.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd96d1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd40c46a0 0x7fbdd41f3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:06.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd96d1640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdd40c4a80 0x7fbdd41f5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:06.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd96d1640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbdd4157180 0x7fbdd41f58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:06.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd96d1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd41496e0 con 0x7fbdd40c46a0 2023-12-04T05:40:06.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd96d1640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd4142be0 con 0x7fbdd40c4a80 2023-12-04T05:40:06.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd96d1640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd4142ea0 con 0x7fbdd4157180 2023-12-04T05:40:06.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd40c46a0 0x7fbdd41f3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:06.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd40c46a0 0x7fbdd41f3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33940/0 (socket says 172.21.15.107:33940) 2023-12-04T05:40:06.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd2ffd640 1 -- 172.21.15.107:0/1447101083 learned_addr learned my addr 172.21.15.107:0/1447101083 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:06.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd2ffd640 1 -- 172.21.15.107:0/1447101083 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbdd4157180 msgr2=0x7fbdd41f58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:06.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd2ffd640 1 --2- 172.21.15.107:0/1447101083 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbdd4157180 0x7fbdd41f58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd2ffd640 1 -- 172.21.15.107:0/1447101083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdd40c4a80 msgr2=0x7fbdd41f5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:06.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd2ffd640 1 --2- 172.21.15.107:0/1447101083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdd40c4a80 0x7fbdd41f5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd2ffd640 1 -- 172.21.15.107:0/1447101083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdcc009b30 con 0x7fbdd40c46a0 2023-12-04T05:40:06.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd2ffd640 1 --2- 172.21.15.107:0/1447101083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd40c46a0 0x7fbdd41f3290 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbdc400c500 tx=0x7fbdc400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:06.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdcb7fe640 1 -- 172.21.15.107:0/1447101083 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4005220 con 0x7fbdd40c46a0 2023-12-04T05:40:06.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdcb7fe640 1 -- 172.21.15.107:0/1447101083 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc4005380 con 0x7fbdd40c46a0 2023-12-04T05:40:06.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd96d1640 1 -- 172.21.15.107:0/1447101083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd414b020 con 0x7fbdd40c46a0 2023-12-04T05:40:06.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdcb7fe640 1 -- 172.21.15.107:0/1447101083 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4007270 con 0x7fbdd40c46a0 2023-12-04T05:40:06.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.659+0000 7fbdd96d1640 1 -- 172.21.15.107:0/1447101083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd414b5e0 con 0x7fbdd40c46a0 2023-12-04T05:40:06.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.663+0000 7fbdd96d1640 1 -- 172.21.15.107:0/1447101083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd4067870 con 0x7fbdd40c46a0 2023-12-04T05:40:06.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.663+0000 7fbdcb7fe640 1 -- 172.21.15.107:0/1447101083 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fbdc40073d0 con 0x7fbdd40c46a0 2023-12-04T05:40:06.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.663+0000 7fbdcb7fe640 1 --2- 172.21.15.107:0/1447101083 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbdb4066900 0x7fbdb4068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:06.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.663+0000 7fbdcb7fe640 1 -- 172.21.15.107:0/1447101083 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbdc4083b60 con 0x7fbdd40c46a0 2023-12-04T05:40:06.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.663+0000 7fbdcbfff640 1 --2- 172.21.15.107:0/1447101083 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbdb4066900 0x7fbdb4068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:06.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.663+0000 7fbdcbfff640 1 --2- 172.21.15.107:0/1447101083 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbdb4066900 0x7fbdb4068dc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbdd41508a0 tx=0x7fbdbc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:06.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.667+0000 7fbdcb7fe640 1 -- 172.21.15.107:0/1447101083 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbdc4050db0 con 0x7fbdd40c46a0 2023-12-04T05:40:06.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.823+0000 7fbdd96d1640 1 -- 172.21.15.107:0/1447101083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbdd406e660 con 0x7fbdd40c46a0 2023-12-04T05:40:06.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.827+0000 7fbdcb7fe640 1 -- 172.21.15.107:0/1447101083 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7fbdd406e660 con 0x7fbdd40c46a0 2023-12-04T05:40:06.828 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:06.828 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":21,"fsid":"603e856b-31ac-475a-8f35-21ab7faadd48","created":"2023-12-04T05:39:34.921690+0000","modified":"2023-12-04T05:40:00.671650+0000","last_up_change":"2023-12-04T05:39:46.235790+0000","last_in_change":"2023-12-04T05:39:37.587639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T05:39:47.567702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T05:39:57.644286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T05:39:58.029897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ab7111c-0ae8-4d41-b6db-e3743614322c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6817","nonce":768714279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6821","nonce":768714279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6829","nonce":768714279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6825","nonce":768714279}]},"public_addr":"172.21.15.107:6817/768714279","cluster_addr":"172.21.15.107:6821/768714279","heartbeat_back_addr":"172.21.15.107:6829/768714279","heartbeat_front_addr":"172.21.15.107:6825/768714279","state":["exists","up"]},{"osd":1,"uuid":"baef7a1f-8026-4fab-9e70-48465271034e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6801","nonce":2496355981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6803","nonce":2496355981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6807","nonce":2496355981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6805","nonce":2496355981}]},"public_addr":"172.21.15.107:6801/2496355981","cluster_addr":"172.21.15.107:6803/2496355981","heartbeat_back_addr":"172.21.15.107:6807/2496355981","heartbeat_front_addr":"172.21.15.107:6805/2496355981","state":["exists","up"]},{"osd":2,"uuid":"90e8cf24-7b54-484a-9d85-b44a9d3dfeab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6809","nonce":3097298224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6811","nonce":3097298224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6815","nonce":3097298224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6813","nonce":3097298224}]},"public_addr":"172.21.15.107:6809/3097298224","cluster_addr":"172.21.15.107:6811/3097298224","heartbeat_back_addr":"172.21.15.107:6815/3097298224","heartbeat_front_addr":"172.21.15.107:6813/3097298224","state":["exists","up"]},{"osd":3,"uuid":"54658731-2b9a-4d5e-ab13-7c1be7d34e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6820","nonce":3773076671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6824","nonce":3773076671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6831","nonce":3773076671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6828","nonce":3773076671}]},"public_addr":"172.21.15.107:6820/3773076671","cluster_addr":"172.21.15.107:6824/3773076671","heartbeat_back_addr":"172.21.15.107:6831/3773076671","heartbeat_front_addr":"172.21.15.107:6828/3773076671","state":["exists","up"]},{"osd":4,"uuid":"9a9dea13-8afe-485d-aa43-45eda691179b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6801","nonce":2747823542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6803","nonce":2747823542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6807","nonce":2747823542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6805","nonce":2747823542}]},"public_addr":"172.21.15.195:6801/2747823542","cluster_addr":"172.21.15.195:6803/2747823542","heartbeat_back_addr":"172.21.15.195:6807/2747823542","heartbeat_front_addr":"172.21.15.195:6805/2747823542","state":["exists","up"]},{"osd":5,"uuid":"db840d83-5a7d-468a-8aa2-1997945847c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6825","nonce":1484264540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6827","nonce":1484264540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6831","nonce":1484264540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6829","nonce":1484264540}]},"public_addr":"172.21.15.195:6825/1484264540","cluster_addr":"172.21.15.195:6827/1484264540","heartbeat_back_addr":"172.21.15.195:6831/1484264540","heartbeat_front_addr":"172.21.15.195:6829/1484264540","state":["exists","up"]},{"osd":6,"uuid":"38b5f165-bf91-467e-a615-4b4784337689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6811","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6813","nonce":2638471412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6817","nonce":2638471412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6823","nonce":2638471412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6821","nonce":2638471412}]},"public_addr":"172.21.15.195:6813/2638471412","cluster_addr":"172.21.15.195:6817/2638471412","heartbeat_back_addr":"172.21.15.195:6823/2638471412","heartbeat_front_addr":"172.21.15.195:6821/2638471412","state":["exists","up"]},{"osd":7,"uuid":"e5b3c596-494b-4df3-9437-bc2d52168ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6809","nonce":768189614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6812","nonce":768189614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6820","nonce":768189614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6816","nonce":768189614}]},"public_addr":"172.21.15.195:6809/768189614","cluster_addr":"172.21.15.195:6812/768189614","heartbeat_back_addr":"172.21.15.195:6820/768189614","heartbeat_front_addr":"172.21.15.195:6816/768189614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.209077+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.086601+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.221568+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.238524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.059477+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.148422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.079867+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.064916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T05:40:06.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.827+0000 7fbdd96d1640 1 -- 172.21.15.107:0/1447101083 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbdb4066900 msgr2=0x7fbdb4068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:06.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.827+0000 7fbdd96d1640 1 --2- 172.21.15.107:0/1447101083 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbdb4066900 0x7fbdb4068dc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbdd41508a0 tx=0x7fbdbc008040 comp rx=0 tx=0).stop 2023-12-04T05:40:06.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.827+0000 7fbdd96d1640 1 -- 172.21.15.107:0/1447101083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd40c46a0 msgr2=0x7fbdd41f3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:06.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.827+0000 7fbdd96d1640 1 --2- 172.21.15.107:0/1447101083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd40c46a0 0x7fbdd41f3290 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbdc400c500 tx=0x7fbdc400c9c0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.827+0000 7fbdd96d1640 1 -- 172.21.15.107:0/1447101083 shutdown_connections 2023-12-04T05:40:06.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.827+0000 7fbdd96d1640 1 --2- 172.21.15.107:0/1447101083 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbdb4066900 0x7fbdb4068dc0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.827+0000 7fbdd96d1640 1 --2- 172.21.15.107:0/1447101083 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbdd4157180 0x7fbdd41f58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.827+0000 7fbdd96d1640 1 --2- 172.21.15.107:0/1447101083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdd40c4a80 0x7fbdd41f5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.827+0000 7fbdd96d1640 1 --2- 172.21.15.107:0/1447101083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd40c46a0 0x7fbdd41f3290 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.827+0000 7fbdd96d1640 1 -- 172.21.15.107:0/1447101083 >> 172.21.15.107:0/1447101083 conn(0x7fbdd40c7ff0 msgr2=0x7fbdd40717b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:06.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.827+0000 7fbdd96d1640 1 -- 172.21.15.107:0/1447101083 shutdown_connections 2023-12-04T05:40:06.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.827+0000 7fbdd96d1640 1 -- 172.21.15.107:0/1447101083 wait complete. 2023-12-04T05:40:06.846 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-04T05:40:06.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T05:40:06.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.963+0000 7f3cd68b5640 1 Processor -- start 2023-12-04T05:40:06.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.963+0000 7f3cd68b5640 1 -- start start 2023-12-04T05:40:06.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.963+0000 7f3cd68b5640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3cd0159f40 0x7f3cd0153da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:06.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.963+0000 7f3cd68b5640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3cd006abf0 0x7f3cd01542e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:06.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.963+0000 7f3cd68b5640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd00c7430 con 0x7f3cd006a810 2023-12-04T05:40:06.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.963+0000 7f3cd68b5640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd0072680 con 0x7f3cd006abf0 2023-12-04T05:40:06.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.963+0000 7f3cd68b5640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd00c8d00 con 0x7f3cd0159f40 2023-12-04T05:40:06.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.963+0000 7f3ccf7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3cd0159f40 0x7f3cd0153da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:06.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.963+0000 7f3cd4e52640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3cd006abf0 0x7f3cd01542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:06.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.963+0000 7f3ccf7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3cd0159f40 0x7f3cd0153da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.107:56556/0 (socket says 172.21.15.107:56556) 2023-12-04T05:40:06.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.963+0000 7f3ccf7fe640 1 -- 172.21.15.107:0/2799270514 learned_addr learned my addr 172.21.15.107:0/2799270514 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:06.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.963+0000 7f3ccf7fe640 1 -- 172.21.15.107:0/2799270514 >> v2:172.21.15.195:3300/0 conn(0x7f3cd006abf0 msgr2=0x7f3cd01542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:06.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.963+0000 7f3ccf7fe640 1 --2- 172.21.15.107:0/2799270514 >> v2:172.21.15.195:3300/0 conn(0x7f3cd006abf0 0x7f3cd01542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.963+0000 7f3ccf7fe640 1 -- 172.21.15.107:0/2799270514 >> v1:172.21.15.195:6789/0 conn(0x7f3cd006a810 legacy=0x7f3cd00c6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:06.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.963+0000 7f3ccf7fe640 1 -- 172.21.15.107:0/2799270514 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd015a3d0 con 0x7f3cd0159f40 2023-12-04T05:40:06.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.967+0000 7f3ccf7fe640 1 --2- 172.21.15.107:0/2799270514 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3cd0159f40 0x7f3cd0153da0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4009870 tx=0x7f3cc401c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dab63e44fa8ebfc7 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:06.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.967+0000 7f3cceffd640 1 -- 172.21.15.107:0/2799270514 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc40a9400 con 0x7f3cd0159f40 2023-12-04T05:40:06.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.967+0000 7f3cceffd640 1 -- 172.21.15.107:0/2799270514 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc40a9560 con 0x7f3cd0159f40 2023-12-04T05:40:06.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.967+0000 7f3cceffd640 1 -- 172.21.15.107:0/2799270514 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc40a98f0 con 0x7f3cd0159f40 2023-12-04T05:40:06.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.967+0000 7f3cd68b5640 1 -- 172.21.15.107:0/2799270514 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3cd0159f40 msgr2=0x7f3cd0153da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:06.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.967+0000 7f3cd68b5640 1 --2- 172.21.15.107:0/2799270514 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3cd0159f40 0x7f3cd0153da0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4009870 tx=0x7f3cc401c540 comp rx=0 tx=0).stop 2023-12-04T05:40:06.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.967+0000 7f3cd68b5640 1 -- 172.21.15.107:0/2799270514 shutdown_connections 2023-12-04T05:40:06.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.967+0000 7f3cd68b5640 1 --2- 172.21.15.107:0/2799270514 >> v2:172.21.15.195:3300/0 conn(0x7f3cd006abf0 0x7f3cd01542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.967+0000 7f3cd68b5640 1 --2- 172.21.15.107:0/2799270514 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3cd0159f40 0x7f3cd0153da0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.967+0000 7f3cd68b5640 1 -- 172.21.15.107:0/2799270514 >> 172.21.15.107:0/2799270514 conn(0x7f3cd00cc440 msgr2=0x7f3cd00cc850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:06.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.967+0000 7f3cd68b5640 1 -- 172.21.15.107:0/2799270514 shutdown_connections 2023-12-04T05:40:06.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.967+0000 7f3cd68b5640 1 -- 172.21.15.107:0/2799270514 wait complete. 2023-12-04T05:40:06.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.967+0000 7f3cd68b5640 1 Processor -- start 2023-12-04T05:40:06.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.967+0000 7f3cd68b5640 1 -- start start 2023-12-04T05:40:06.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3cd68b5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd006a810 0x7f3cd015bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:06.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3cd68b5640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3cd006abf0 0x7f3cd015c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:06.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3cd68b5640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3cd014e080 0x7f3cd0146660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:06.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3cd68b5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd015dc70 con 0x7f3cd006a810 2023-12-04T05:40:06.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3cd68b5640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd015b1a0 con 0x7f3cd006abf0 2023-12-04T05:40:06.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3cd68b5640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd01f35f0 con 0x7f3cd014e080 2023-12-04T05:40:06.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3ccffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd006a810 0x7f3cd015bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:06.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3ccffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd006a810 0x7f3cd015bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33948/0 (socket says 172.21.15.107:33948) 2023-12-04T05:40:06.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3ccffff640 1 -- 172.21.15.107:0/2428408070 learned_addr learned my addr 172.21.15.107:0/2428408070 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:06.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3cd4e52640 1 --2- 172.21.15.107:0/2428408070 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3cd014e080 0x7f3cd0146660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:06.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3ccf7fe640 1 --2- 172.21.15.107:0/2428408070 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3cd006abf0 0x7f3cd015c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:06.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3ccffff640 1 -- 172.21.15.107:0/2428408070 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3cd014e080 msgr2=0x7f3cd0146660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:06.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3ccffff640 1 --2- 172.21.15.107:0/2428408070 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3cd014e080 0x7f3cd0146660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3ccffff640 1 -- 172.21.15.107:0/2428408070 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3cd006abf0 msgr2=0x7f3cd015c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:06.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3ccffff640 1 --2- 172.21.15.107:0/2428408070 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3cd006abf0 0x7f3cd015c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:06.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3ccffff640 1 -- 172.21.15.107:0/2428408070 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc40094d0 con 0x7f3cd006a810 2023-12-04T05:40:06.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3ccffff640 1 --2- 172.21.15.107:0/2428408070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd006a810 0x7f3cd015bf20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000c370 tx=0x7f3cc000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:06.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3cccff9640 1 -- 172.21.15.107:0/2428408070 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc00052d0 con 0x7f3cd006a810 2023-12-04T05:40:06.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3cccff9640 1 -- 172.21.15.107:0/2428408070 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc0005430 con 0x7f3cd006a810 2023-12-04T05:40:06.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3cd68b5640 1 -- 172.21.15.107:0/2428408070 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd015b440 con 0x7f3cd006a810 2023-12-04T05:40:06.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3cccff9640 1 -- 172.21.15.107:0/2428408070 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc0007260 con 0x7f3cd006a810 2023-12-04T05:40:06.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3cd68b5640 1 -- 172.21.15.107:0/2428408070 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd015ba00 con 0x7f3cd006a810 2023-12-04T05:40:06.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3cd68b5640 1 -- 172.21.15.107:0/2428408070 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd015bc20 con 0x7f3cd006a810 2023-12-04T05:40:06.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3cccff9640 1 -- 172.21.15.107:0/2428408070 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f3cc00073c0 con 0x7f3cd006a810 2023-12-04T05:40:06.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3cccff9640 1 --2- 172.21.15.107:0/2428408070 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3cbc066900 0x7f3cbc068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:06.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3cccff9640 1 -- 172.21.15.107:0/2428408070 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3cc0083820 con 0x7f3cd006a810 2023-12-04T05:40:06.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.971+0000 7f3ccf7fe640 1 --2- 172.21.15.107:0/2428408070 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3cbc066900 0x7f3cbc068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:06.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.975+0000 7f3ccf7fe640 1 --2- 172.21.15.107:0/2428408070 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3cbc066900 0x7f3cbc068dc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3cc40ae000 tx=0x7f3cc40b3040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:06.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:06.979+0000 7f3cccff9640 1 -- 172.21.15.107:0/2428408070 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3cc00509d0 con 0x7f3cd006a810 2023-12-04T05:40:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.131+0000 7f3cd68b5640 1 -- 172.21.15.107:0/2428408070 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3cd0067ec0 con 0x7f3cd006a810 2023-12-04T05:40:07.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.135+0000 7f3cccff9640 1 -- 172.21.15.107:0/2428408070 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f3cc0083020 con 0x7f3cd006a810 2023-12-04T05:40:07.138 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:07.138 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":21,"fsid":"603e856b-31ac-475a-8f35-21ab7faadd48","created":"2023-12-04T05:39:34.921690+0000","modified":"2023-12-04T05:40:00.671650+0000","last_up_change":"2023-12-04T05:39:46.235790+0000","last_in_change":"2023-12-04T05:39:37.587639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T05:39:47.567702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T05:39:57.644286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T05:39:58.029897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ab7111c-0ae8-4d41-b6db-e3743614322c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6817","nonce":768714279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6821","nonce":768714279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6829","nonce":768714279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6825","nonce":768714279}]},"public_addr":"172.21.15.107:6817/768714279","cluster_addr":"172.21.15.107:6821/768714279","heartbeat_back_addr":"172.21.15.107:6829/768714279","heartbeat_front_addr":"172.21.15.107:6825/768714279","state":["exists","up"]},{"osd":1,"uuid":"baef7a1f-8026-4fab-9e70-48465271034e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6801","nonce":2496355981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6803","nonce":2496355981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6807","nonce":2496355981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6805","nonce":2496355981}]},"public_addr":"172.21.15.107:6801/2496355981","cluster_addr":"172.21.15.107:6803/2496355981","heartbeat_back_addr":"172.21.15.107:6807/2496355981","heartbeat_front_addr":"172.21.15.107:6805/2496355981","state":["exists","up"]},{"osd":2,"uuid":"90e8cf24-7b54-484a-9d85-b44a9d3dfeab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6809","nonce":3097298224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6811","nonce":3097298224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6815","nonce":3097298224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6813","nonce":3097298224}]},"public_addr":"172.21.15.107:6809/3097298224","cluster_addr":"172.21.15.107:6811/3097298224","heartbeat_back_addr":"172.21.15.107:6815/3097298224","heartbeat_front_addr":"172.21.15.107:6813/3097298224","state":["exists","up"]},{"osd":3,"uuid":"54658731-2b9a-4d5e-ab13-7c1be7d34e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6820","nonce":3773076671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6824","nonce":3773076671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6831","nonce":3773076671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6828","nonce":3773076671}]},"public_addr":"172.21.15.107:6820/3773076671","cluster_addr":"172.21.15.107:6824/3773076671","heartbeat_back_addr":"172.21.15.107:6831/3773076671","heartbeat_front_addr":"172.21.15.107:6828/3773076671","state":["exists","up"]},{"osd":4,"uuid":"9a9dea13-8afe-485d-aa43-45eda691179b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6801","nonce":2747823542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6803","nonce":2747823542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6807","nonce":2747823542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6805","nonce":2747823542}]},"public_addr":"172.21.15.195:6801/2747823542","cluster_addr":"172.21.15.195:6803/2747823542","heartbeat_back_addr":"172.21.15.195:6807/2747823542","heartbeat_front_addr":"172.21.15.195:6805/2747823542","state":["exists","up"]},{"osd":5,"uuid":"db840d83-5a7d-468a-8aa2-1997945847c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6825","nonce":1484264540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6827","nonce":1484264540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6831","nonce":1484264540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6829","nonce":1484264540}]},"public_addr":"172.21.15.195:6825/1484264540","cluster_addr":"172.21.15.195:6827/1484264540","heartbeat_back_addr":"172.21.15.195:6831/1484264540","heartbeat_front_addr":"172.21.15.195:6829/1484264540","state":["exists","up"]},{"osd":6,"uuid":"38b5f165-bf91-467e-a615-4b4784337689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6811","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6813","nonce":2638471412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6817","nonce":2638471412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6823","nonce":2638471412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6821","nonce":2638471412}]},"public_addr":"172.21.15.195:6813/2638471412","cluster_addr":"172.21.15.195:6817/2638471412","heartbeat_back_addr":"172.21.15.195:6823/2638471412","heartbeat_front_addr":"172.21.15.195:6821/2638471412","state":["exists","up"]},{"osd":7,"uuid":"e5b3c596-494b-4df3-9437-bc2d52168ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6809","nonce":768189614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6812","nonce":768189614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6820","nonce":768189614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6816","nonce":768189614}]},"public_addr":"172.21.15.195:6809/768189614","cluster_addr":"172.21.15.195:6812/768189614","heartbeat_back_addr":"172.21.15.195:6820/768189614","heartbeat_front_addr":"172.21.15.195:6816/768189614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.209077+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.086601+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.221568+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.238524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.059477+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.148422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.079867+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.064916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T05:40:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.139+0000 7f3cd68b5640 1 -- 172.21.15.107:0/2428408070 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3cbc066900 msgr2=0x7f3cbc068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.139+0000 7f3cd68b5640 1 --2- 172.21.15.107:0/2428408070 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3cbc066900 0x7f3cbc068dc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3cc40ae000 tx=0x7f3cc40b3040 comp rx=0 tx=0).stop 2023-12-04T05:40:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.139+0000 7f3cd68b5640 1 -- 172.21.15.107:0/2428408070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd006a810 msgr2=0x7f3cd015bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.139+0000 7f3cd68b5640 1 --2- 172.21.15.107:0/2428408070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd006a810 0x7f3cd015bf20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000c370 tx=0x7f3cc000c830 comp rx=0 tx=0).stop 2023-12-04T05:40:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.139+0000 7f3cd68b5640 1 -- 172.21.15.107:0/2428408070 shutdown_connections 2023-12-04T05:40:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.139+0000 7f3cd68b5640 1 --2- 172.21.15.107:0/2428408070 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3cbc066900 0x7f3cbc068dc0 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3cc40ae000 tx=0x7f3cc40b3040 comp rx=0 tx=0).stop 2023-12-04T05:40:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.139+0000 7f3cd68b5640 1 --2- 172.21.15.107:0/2428408070 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3cd014e080 0x7f3cd0146660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.139+0000 7f3cd68b5640 1 --2- 172.21.15.107:0/2428408070 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3cd006abf0 0x7f3cd015c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.139+0000 7f3cd68b5640 1 --2- 172.21.15.107:0/2428408070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd006a810 0x7f3cd015bf20 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.139+0000 7f3cd68b5640 1 -- 172.21.15.107:0/2428408070 >> 172.21.15.107:0/2428408070 conn(0x7f3cd00cc440 msgr2=0x7f3cd014a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.139+0000 7f3cd68b5640 1 -- 172.21.15.107:0/2428408070 shutdown_connections 2023-12-04T05:40:07.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.139+0000 7f3cd68b5640 1 -- 172.21.15.107:0/2428408070 wait complete. 2023-12-04T05:40:07.157 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-04T05:40:07.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-04T05:40:07.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-04T05:40:07.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-12-04T05:40:07.159 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-12-04T05:40:07.159 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-12-04T05:40:07.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-12-04T05:40:07.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-12-04T05:40:07.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.335+0000 7f6c75d19640 1 Processor -- start 2023-12-04T05:40:07.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.335+0000 7f18fa74a640 1 Processor -- start 2023-12-04T05:40:07.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.335+0000 7f18fa74a640 1 -- start start 2023-12-04T05:40:07.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.335+0000 7f6c75d19640 1 -- start start 2023-12-04T05:40:07.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7fec5f993640 1 Processor -- start 2023-12-04T05:40:07.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f18fa74a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f18f4151fb0 0x7f18f40724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f18fa74a640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f18f40c9a10 0x7f18f40ca450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f18fa74a640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18f40c6ba0 con 0x7f18f40ca990 2023-12-04T05:40:07.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f18fa74a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18f4157da0 con 0x7f18f4151fb0 2023-12-04T05:40:07.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f18fa74a640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f40c65c0 con 0x7f18f40c9a10 2023-12-04T05:40:07.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f6c75d19640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6c70159830 0x7f6c7006df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f6c75d19640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6c700c7500 0x7f6c70159c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f6c75d19640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6c7015a150 0x7f6c7014f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f6c75d19640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c700c7d00 con 0x7f6c7015a150 2023-12-04T05:40:07.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f6c75d19640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c7006da80 con 0x7f6c700c7500 2023-12-04T05:40:07.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f6c75d19640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c70071940 con 0x7f6c70159830 2023-12-04T05:40:07.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f18ebfff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f18f40c9a10 0x7f18f40ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f18f8ce7640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f18f40ca990 0x7f18f41581c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:37590/0 (socket says 172.21.15.107:37590) 2023-12-04T05:40:07.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f18f8ce7640 1 -- 172.21.15.107:0/3966870790 learned_addr learned my addr 172.21.15.107:0/3966870790 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:07.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f6c6ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6c7015a150 0x7f6c7014f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f6c6ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6c7015a150 0x7f6c7014f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33954/0 (socket says 172.21.15.107:33954) 2023-12-04T05:40:07.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f6c6ffff640 1 -- 172.21.15.107:0/3477601871 learned_addr learned my addr 172.21.15.107:0/3477601871 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f6c6effd640 1 --2- 172.21.15.107:0/3477601871 >> v2:172.21.15.195:3300/0 conn(0x7f6c700c7500 0x7f6c70159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f18f3fff640 1 --2- 172.21.15.107:0/3966870790 >> v2:172.21.15.107:3300/0 conn(0x7f18f4151fb0 0x7f18f40724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f6c6f7fe640 1 --2- 172.21.15.107:0/3477601871 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6c70159830 0x7f6c7006df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f18f37fe640 1 -- 172.21.15.107:0/3966870790 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2328172661 0 0) 0x7f18f40c6ba0 con 0x7f18f40ca990 2023-12-04T05:40:07.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f18f37fe640 1 -- 172.21.15.107:0/3966870790 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18e0003530 con 0x7f18f40ca990 2023-12-04T05:40:07.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f7e2a14d640 1 Processor -- start 2023-12-04T05:40:07.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7fec5f993640 1 -- start start 2023-12-04T05:40:07.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7fec5f993640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fec58150dc0 0x7fec5815a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7fec5f993640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fec58071080 0x7fec5815ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7fec5f993640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5815a390 con 0x7fec58156b70 2023-12-04T05:40:07.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7fec5f993640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec58157ba0 con 0x7fec58071080 2023-12-04T05:40:07.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7fec5f993640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec58157040 con 0x7fec58150dc0 2023-12-04T05:40:07.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7fec5cf2e640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fec58156b70 0x7fec58071740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:37606/0 (socket says 172.21.15.107:37606) 2023-12-04T05:40:07.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7fec5cf2e640 1 -- 172.21.15.107:0/550784067 learned_addr learned my addr 172.21.15.107:0/550784067 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:07.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7fec5df30640 1 --2- 172.21.15.107:0/550784067 >> v2:172.21.15.195:3300/0 conn(0x7fec58071080 0x7fec5815ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f7e2a14d640 1 -- start start 2023-12-04T05:40:07.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f7e2a14d640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f7e24159830 0x7f7e2406df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f7e2a14d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e24155fe0 0x7f7e24156470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f7e2a14d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2406da80 con 0x7f7e240c7500 2023-12-04T05:40:07.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f7e2a14d640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e240c7d00 con 0x7f7e24159830 2023-12-04T05:40:07.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f7e2a14d640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e24071940 con 0x7f7e24155fe0 2023-12-04T05:40:07.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f18f3fff640 1 -- 172.21.15.107:0/3966870790 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f18f40c9a10 msgr2=0x7f18f40ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:07.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f18f3fff640 1 --2- 172.21.15.107:0/3966870790 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f18f40c9a10 0x7f18f40ca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f18f3fff640 1 -- 172.21.15.107:0/3966870790 >> v1:172.21.15.195:6789/0 conn(0x7f18f40ca990 legacy=0x7f18f41581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f18f3fff640 1 -- 172.21.15.107:0/3966870790 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f4072e80 con 0x7f18f4151fb0 2023-12-04T05:40:07.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f6c6ffff640 1 -- 172.21.15.107:0/3477601871 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6c70159830 msgr2=0x7f6c7006df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:07.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f6c6ffff640 1 --2- 172.21.15.107:0/3477601871 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6c70159830 0x7f6c7006df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f6c6ffff640 1 -- 172.21.15.107:0/3477601871 >> v2:172.21.15.195:3300/0 conn(0x7f6c700c7500 msgr2=0x7f6c70159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:07.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f6c6ffff640 1 --2- 172.21.15.107:0/3477601871 >> v2:172.21.15.195:3300/0 conn(0x7f6c700c7500 0x7f6c70159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f6c6ffff640 1 -- 172.21.15.107:0/3477601871 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c700713c0 con 0x7f6c7015a150 2023-12-04T05:40:07.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f7e23fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e24155fe0 0x7f7e24156470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f7e23fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e24155fe0 0x7f7e24156470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.107:56602/0 (socket says 172.21.15.107:56602) 2023-12-04T05:40:07.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7f7e23fff640 1 -- 172.21.15.107:0/4242013523 learned_addr learned my addr 172.21.15.107:0/4242013523 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7fec5df30640 1 -- 172.21.15.107:0/550784067 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fec58150dc0 msgr2=0x7fec5815a820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-12-04T05:40:07.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7fec5df30640 1 --2- 172.21.15.107:0/550784067 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fec58150dc0 0x7fec5815a820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7fec5df30640 1 -- 172.21.15.107:0/550784067 >> v1:172.21.15.195:6789/0 conn(0x7fec58156b70 legacy=0x7fec58071740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.339+0000 7fec5df30640 1 -- 172.21.15.107:0/550784067 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec58157300 con 0x7fec58071080 2023-12-04T05:40:07.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f7e237fe640 1 --2- 172.21.15.107:0/4242013523 >> v2:172.21.15.195:3300/0 conn(0x7f7e24159830 0x7f7e2406df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f415884a640 1 Processor -- start 2023-12-04T05:40:07.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f415884a640 1 -- start start 2023-12-04T05:40:07.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f7e23fff640 1 -- 172.21.15.107:0/4242013523 >> v2:172.21.15.195:3300/0 conn(0x7f7e24159830 msgr2=0x7f7e2406df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:07.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f7e23fff640 1 --2- 172.21.15.107:0/4242013523 >> v2:172.21.15.195:3300/0 conn(0x7f7e24159830 0x7f7e2406df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f7e23fff640 1 -- 172.21.15.107:0/4242013523 >> v1:172.21.15.107:6789/0 conn(0x7f7e240c7500 legacy=0x7f7e24159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f7e23fff640 1 -- 172.21.15.107:0/4242013523 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e240713c0 con 0x7f7e24155fe0 2023-12-04T05:40:07.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f18f3fff640 1 --2- 172.21.15.107:0/3966870790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18f4151fb0 0x7f18f40724c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f18e40097c0 tx=0x7f18e401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a74e4f45ef36750a server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7ff53de7c640 1 Processor -- start 2023-12-04T05:40:07.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7ff53de7c640 1 -- start start 2023-12-04T05:40:07.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f6c6ffff640 1 --2- 172.21.15.107:0/3477601871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c7015a150 0x7f6c7014f920 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6c7006ba50 tx=0x7f6c64021210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b059c08f677dda10 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f18f37fe640 1 -- 172.21.15.107:0/3966870790 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e40ad640 con 0x7f18f4151fb0 2023-12-04T05:40:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f6c6e7fc640 1 -- 172.21.15.107:0/3477601871 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c640b3410 con 0x7f6c7015a150 2023-12-04T05:40:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7fec5df30640 1 --2- 172.21.15.107:0/550784067 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec58071080 0x7fec5815ad60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fec50009d90 tx=0x7fec5001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10fc810d9f07897a server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f415884a640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f414c04e130 0x7f414c0dd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f18f37fe640 1 -- 172.21.15.107:0/3966870790 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e40ad7a0 con 0x7f18f4151fb0 2023-12-04T05:40:07.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f18f37fe640 1 -- 172.21.15.107:0/3966870790 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e40adb30 con 0x7f18f4151fb0 2023-12-04T05:40:07.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f415884a640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f414c04d170 0x7f414c04ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f217ac16640 1 Processor -- start 2023-12-04T05:40:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f6c6e7fc640 1 -- 172.21.15.107:0/3477601871 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c640ad030 con 0x7f6c7015a150 2023-12-04T05:40:07.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f6c6e7fc640 1 -- 172.21.15.107:0/3477601871 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c640b3790 con 0x7f6c7015a150 2023-12-04T05:40:07.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f415884a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f414c0ddc40 con 0x7f414c0d7060 2023-12-04T05:40:07.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f4152d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f414c0d7060 0x7f414c0e0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60356/0 (socket says 172.21.15.107:60356) 2023-12-04T05:40:07.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f4152d9d640 1 -- 172.21.15.107:0/823542579 learned_addr learned my addr 172.21.15.107:0/823542579 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:07.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f415884a640 1 -- 172.21.15.107:0/823542579 --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f414c0d7440 con 0x7f414c04e130 2023-12-04T05:40:07.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f415884a640 1 -- 172.21.15.107:0/823542579 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f414c04d740 con 0x7f414c04d170 2023-12-04T05:40:07.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f415259c640 1 --2- 172.21.15.107:0/823542579 >> v2:172.21.15.195:3300/0 conn(0x7f414c04e130 0x7f414c0dd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f415904b640 1 --2- 172.21.15.107:0/823542579 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f414c04d170 0x7f414c04ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f7e49010640 1 Processor -- start 2023-12-04T05:40:07.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f7e49010640 1 -- start start 2023-12-04T05:40:07.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7ff53de7c640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff5300d8ff0 0x7ff53004aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f18fa74a640 1 -- 172.21.15.107:0/3966870790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18f4151fb0 msgr2=0x7f18f40724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f18fa74a640 1 --2- 172.21.15.107:0/3966870790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18f4151fb0 0x7f18f40724c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f18e40097c0 tx=0x7f18e401c4a0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7ff53de7c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff530046780 0x7ff5300e22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f217ac16640 1 -- start start 2023-12-04T05:40:07.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f217ac16640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2174159830 0x7f217406df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7ff53de7c640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff530006280 con 0x7ff530005e70 2023-12-04T05:40:07.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f6c75d19640 1 -- 172.21.15.107:0/3477601871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c7015a150 msgr2=0x7f6c7014f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f6c75d19640 1 --2- 172.21.15.107:0/3477601871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c7015a150 0x7f6c7014f920 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6c7006ba50 tx=0x7f6c64021210 comp rx=0 tx=0).stop 2023-12-04T05:40:07.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f217ac16640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2174155fe0 0x7f2174156470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f217ac16640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f217406da80 con 0x7f21740c7500 2023-12-04T05:40:07.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f217ac16640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21740c7d00 con 0x7f2174155fe0 2023-12-04T05:40:07.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f217ac16640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2174071940 con 0x7f2174159830 2023-12-04T05:40:07.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7ff53de7c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff53004b4a0 con 0x7ff530046780 2023-12-04T05:40:07.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7ff537fff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff530005e70 0x7ff53004c450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:37622/0 (socket says 172.21.15.107:37622) 2023-12-04T05:40:07.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7ff537fff640 1 -- 172.21.15.107:0/4008331055 learned_addr learned my addr 172.21.15.107:0/4008331055 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:07.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7ff53de7c640 1 -- 172.21.15.107:0/4008331055 --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5300d93d0 con 0x7ff5300d8ff0 2023-12-04T05:40:07.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f6c75d19640 1 -- 172.21.15.107:0/3477601871 shutdown_connections 2023-12-04T05:40:07.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f6c75d19640 1 --2- 172.21.15.107:0/3477601871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c7015a150 0x7f6c7014f920 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f6c75d19640 1 --2- 172.21.15.107:0/3477601871 >> v2:172.21.15.195:3300/0 conn(0x7f6c700c7500 0x7f6c70159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f6c75d19640 1 --2- 172.21.15.107:0/3477601871 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6c70159830 0x7f6c7006df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f6c75d19640 1 -- 172.21.15.107:0/3477601871 >> 172.21.15.107:0/3477601871 conn(0x7f6c700c7ff0 msgr2=0x7f6c700c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f6c75d19640 1 -- 172.21.15.107:0/3477601871 shutdown_connections 2023-12-04T05:40:07.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7ff53ce7a640 1 --2- 172.21.15.107:0/4008331055 >> v2:172.21.15.195:3300/0 conn(0x7ff5300d8ff0 0x7ff53004aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f21791b3640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2174155fe0 0x7f2174156470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f21789b2640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2174159830 0x7f217406df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f2173fff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f21740c7500 0x7f2174159c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:37626/0 (socket says 172.21.15.107:37626) 2023-12-04T05:40:07.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f21789b2640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2174159830 0x7f217406df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.107:56610/0 (socket says 172.21.15.107:56610) 2023-12-04T05:40:07.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f2173fff640 1 -- 172.21.15.107:0/3502536844 learned_addr learned my addr 172.21.15.107:0/3502536844 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:07.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7ff53d67b640 1 --2- 172.21.15.107:0/4008331055 >> v2:172.21.15.107:3300/0 conn(0x7ff530046780 0x7ff5300e22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f6c75d19640 1 -- 172.21.15.107:0/3477601871 wait complete. 2023-12-04T05:40:07.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7ff5377fe640 1 -- 172.21.15.107:0/4008331055 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557146031 0 0) 0x7ff524001120 con 0x7ff530005e70 2023-12-04T05:40:07.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7ff5377fe640 1 -- 172.21.15.107:0/4008331055 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff528003530 con 0x7ff530005e70 2023-12-04T05:40:07.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e23fff640 1 --2- 172.21.15.107:0/4242013523 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e24155fe0 0x7f7e24156470 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c009e50 tx=0x7f7e1c01ec00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82fa57b977439a04 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e49010640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e44158420 0x7f7e4415ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e49010640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f7e440ca140 0x7f7e4406be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e49010640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e440c4730 con 0x7f7e4406c340 2023-12-04T05:40:07.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e49010640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e44161790 con 0x7f7e440ca140 2023-12-04T05:40:07.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e49010640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e440b23f0 con 0x7f7e44158420 2023-12-04T05:40:07.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7ff53ce7a640 1 -- 172.21.15.107:0/4008331055 >> v2:172.21.15.107:3300/0 conn(0x7ff530046780 msgr2=0x7ff5300e22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:07.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7ff53ce7a640 1 --2- 172.21.15.107:0/4008331055 >> v2:172.21.15.107:3300/0 conn(0x7ff530046780 0x7ff5300e22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7ff53ce7a640 1 -- 172.21.15.107:0/4008331055 >> v1:172.21.15.195:6789/0 conn(0x7ff530005e70 legacy=0x7ff53004c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7ff53ce7a640 1 -- 172.21.15.107:0/4008331055 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5300459c0 con 0x7ff5300d8ff0 2023-12-04T05:40:07.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7ff53d67b640 1 --2- 172.21.15.107:0/4008331055 >> v2:172.21.15.107:3300/0 conn(0x7ff530046780 0x7ff5300e22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T05:40:07.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f21789b2640 1 -- 172.21.15.107:0/3502536844 >> v2:172.21.15.195:3300/0 conn(0x7f2174155fe0 msgr2=0x7f2174156470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:07.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f21789b2640 1 --2- 172.21.15.107:0/3502536844 >> v2:172.21.15.195:3300/0 conn(0x7f2174155fe0 0x7f2174156470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f21789b2640 1 -- 172.21.15.107:0/3502536844 >> v1:172.21.15.195:6789/0 conn(0x7f21740c7500 legacy=0x7f2174159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f21789b2640 1 -- 172.21.15.107:0/3502536844 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21740713c0 con 0x7f2174159830 2023-12-04T05:40:07.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7fec4ffff640 1 -- 172.21.15.107:0/550784067 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec500a5a30 con 0x7fec58071080 2023-12-04T05:40:07.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7fec4ffff640 1 -- 172.21.15.107:0/550784067 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec500a5b90 con 0x7fec58071080 2023-12-04T05:40:07.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7fec4ffff640 1 -- 172.21.15.107:0/550784067 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec500ad440 con 0x7fec58071080 2023-12-04T05:40:07.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7fec5f993640 1 -- 172.21.15.107:0/550784067 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec58071080 msgr2=0x7fec5815ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7fec5f993640 1 --2- 172.21.15.107:0/550784067 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec58071080 0x7fec5815ad60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fec50009d90 tx=0x7fec5001c970 comp rx=0 tx=0).stop 2023-12-04T05:40:07.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7fec5f993640 1 -- 172.21.15.107:0/550784067 shutdown_connections 2023-12-04T05:40:07.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7fec5f993640 1 --2- 172.21.15.107:0/550784067 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec58071080 0x7fec5815ad60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7fec5f993640 1 --2- 172.21.15.107:0/550784067 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fec58150dc0 0x7fec5815a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7fec5f993640 1 -- 172.21.15.107:0/550784067 >> 172.21.15.107:0/550784067 conn(0x7fec580cc490 msgr2=0x7fec580cc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f415904b640 1 -- 172.21.15.107:0/823542579 >> v2:172.21.15.195:3300/0 conn(0x7f414c04e130 msgr2=0x7f414c0dd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f415904b640 1 --2- 172.21.15.107:0/823542579 >> v2:172.21.15.195:3300/0 conn(0x7f414c04e130 0x7f414c0dd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f415904b640 1 -- 172.21.15.107:0/823542579 >> v1:172.21.15.107:6789/0 conn(0x7f414c0d7060 legacy=0x7f414c0e0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.343+0000 7f415904b640 1 -- 172.21.15.107:0/823542579 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f414c0e05e0 con 0x7f414c04d170 2023-12-04T05:40:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f18fa74a640 1 -- 172.21.15.107:0/3966870790 shutdown_connections 2023-12-04T05:40:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f18fa74a640 1 --2- 172.21.15.107:0/3966870790 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f18f40c9a10 0x7f18f40ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f18fa74a640 1 --2- 172.21.15.107:0/3966870790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18f4151fb0 0x7f18f40724c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f18fa74a640 1 -- 172.21.15.107:0/3966870790 >> 172.21.15.107:0/3966870790 conn(0x7f18f4066570 msgr2=0x7f18f4066980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f6c75d19640 1 Processor -- start 2023-12-04T05:40:07.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e227fc640 1 -- 172.21.15.107:0/4242013523 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c0a7ad0 con 0x7f7e24155fe0 2023-12-04T05:40:07.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e227fc640 1 -- 172.21.15.107:0/4242013523 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1c0a7c30 con 0x7f7e24155fe0 2023-12-04T05:40:07.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e2a14d640 1 -- 172.21.15.107:0/4242013523 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e24155fe0 msgr2=0x7f7e24156470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e2a14d640 1 --2- 172.21.15.107:0/4242013523 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e24155fe0 0x7f7e24156470 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c009e50 tx=0x7f7e1c01ec00 comp rx=0 tx=0).stop 2023-12-04T05:40:07.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f18fa74a640 1 -- 172.21.15.107:0/3966870790 shutdown_connections 2023-12-04T05:40:07.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7fec5f993640 1 -- 172.21.15.107:0/550784067 shutdown_connections 2023-12-04T05:40:07.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e2a14d640 1 -- 172.21.15.107:0/4242013523 shutdown_connections 2023-12-04T05:40:07.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e2a14d640 1 --2- 172.21.15.107:0/4242013523 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e24155fe0 0x7f7e24156470 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e2a14d640 1 --2- 172.21.15.107:0/4242013523 >> v2:172.21.15.195:3300/0 conn(0x7f7e24159830 0x7f7e2406df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e2a14d640 1 -- 172.21.15.107:0/4242013523 >> 172.21.15.107:0/4242013523 conn(0x7f7e240c7ff0 msgr2=0x7f7e240c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e2a14d640 1 -- 172.21.15.107:0/4242013523 shutdown_connections 2023-12-04T05:40:07.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f6c75d19640 1 -- start start 2023-12-04T05:40:07.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f6c75d19640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6c700c7500 0x7f6c70157fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f6c75d19640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c70159830 0x7f6c7014e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f6c75d19640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c7015a150 0x7f6c7014ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f6c75d19640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c701e34f0 con 0x7f6c70159830 2023-12-04T05:40:07.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f6c75d19640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c701e3df0 con 0x7f6c7015a150 2023-12-04T05:40:07.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f6c75d19640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c701e31a0 con 0x7f6c700c7500 2023-12-04T05:40:07.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f6c6ffff640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c7015a150 0x7f6c7014ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f6c6f7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6c700c7500 0x7f6c70157fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f6c6ffff640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c7015a150 0x7f6c7014ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.107:51508/0 (socket says 172.21.15.107:51508) 2023-12-04T05:40:07.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f6c6ffff640 1 -- 172.21.15.107:0/3523059291 learned_addr learned my addr 172.21.15.107:0/3523059291 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f18fa74a640 1 -- 172.21.15.107:0/3966870790 wait complete. 2023-12-04T05:40:07.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7fec5f993640 1 -- 172.21.15.107:0/550784067 wait complete. 2023-12-04T05:40:07.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f18fa74a640 1 Processor -- start 2023-12-04T05:40:07.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7fec5f993640 1 Processor -- start 2023-12-04T05:40:07.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7ff53ce7a640 1 --2- 172.21.15.107:0/4008331055 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff5300d8ff0 0x7ff53004aec0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff530036450 tx=0x7ff52c01eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b5f4a1c8c7955a6 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f6c6effd640 1 --2- 172.21.15.107:0/3523059291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c70159830 0x7f6c7014e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e4359e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7e4406c340 0x7f7e440c4bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60362/0 (socket says 172.21.15.107:60362) 2023-12-04T05:40:07.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e4359e640 1 -- 172.21.15.107:0/2833116655 learned_addr learned my addr 172.21.15.107:0/2833116655 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:07.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e4259c640 1 --2- 172.21.15.107:0/2833116655 >> v2:172.21.15.195:3300/0 conn(0x7f7e440ca140 0x7f7e4406be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e4259c640 1 -- 172.21.15.107:0/2833116655 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e44158420 msgr2=0x7f7e4415ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:07.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e4259c640 1 --2- 172.21.15.107:0/2833116655 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e44158420 0x7f7e4415ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e4259c640 1 -- 172.21.15.107:0/2833116655 >> v1:172.21.15.107:6789/0 conn(0x7f7e4406c340 legacy=0x7f7e440c4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e4259c640 1 -- 172.21.15.107:0/2833116655 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e44161a50 con 0x7f7e440ca140 2023-12-04T05:40:07.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7ff5377fe640 1 -- 172.21.15.107:0/4008331055 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c0a8890 con 0x7ff5300d8ff0 2023-12-04T05:40:07.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f7e2a14d640 1 -- 172.21.15.107:0/4242013523 wait complete. 2023-12-04T05:40:07.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e2a14d640 1 Processor -- start 2023-12-04T05:40:07.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f6c6f7fe640 1 -- 172.21.15.107:0/3523059291 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c7015a150 msgr2=0x7f6c7014ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f6c6f7fe640 1 --2- 172.21.15.107:0/3523059291 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6c7015a150 0x7f6c7014ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f6c6f7fe640 1 -- 172.21.15.107:0/3523059291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c70159830 msgr2=0x7f6c7014e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f6c6f7fe640 1 --2- 172.21.15.107:0/3523059291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c70159830 0x7f6c7014e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f6c6f7fe640 1 -- 172.21.15.107:0/3523059291 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c64008550 con 0x7f6c700c7500 2023-12-04T05:40:07.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f415904b640 1 --2- 172.21.15.107:0/823542579 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f414c04d170 0x7f414c04ef50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f41440097c0 tx=0x7f414401c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e85b1566464b21f5 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f4151d9b640 1 -- 172.21.15.107:0/823542579 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41440ab600 con 0x7f414c04d170 2023-12-04T05:40:07.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f4151d9b640 1 -- 172.21.15.107:0/823542579 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41440a7050 con 0x7f414c04d170 2023-12-04T05:40:07.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f4151d9b640 1 -- 172.21.15.107:0/823542579 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41440ab980 con 0x7f414c04d170 2023-12-04T05:40:07.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7fec5f993640 1 -- start start 2023-12-04T05:40:07.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7fec5f993640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fec58071080 0x7fec5806a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7fec5f993640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec58150dc0 0x7fec580c18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7fec5f993640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec58156b70 0x7fec581fb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7fec5f993640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec580c22f0 con 0x7fec58156b70 2023-12-04T05:40:07.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7fec5f993640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec580c25b0 con 0x7fec58150dc0 2023-12-04T05:40:07.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f21789b2640 1 --2- 172.21.15.107:0/3502536844 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2174159830 0x7f217406df10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2168005a20 tx=0x7f216801f5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86930447a1b228c0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7fec5f993640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec580c2890 con 0x7fec58071080 2023-12-04T05:40:07.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.347+0000 7f18fa74a640 1 -- start start 2023-12-04T05:40:07.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f18fa74a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18f40c9a10 0x7f18f41674f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f18fa74a640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f18f40ca990 0x7f18f41f64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7ff5377fe640 1 -- 172.21.15.107:0/4008331055 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff52c0a89f0 con 0x7ff5300d8ff0 2023-12-04T05:40:07.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7ff5377fe640 1 -- 172.21.15.107:0/4008331055 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c0a8d60 con 0x7ff5300d8ff0 2023-12-04T05:40:07.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f18fa74a640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18f4151fb0 0x7f18f4202c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f18fa74a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f4149a00 con 0x7f18f40c9a10 2023-12-04T05:40:07.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f18fa74a640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f414a310 con 0x7f18f4151fb0 2023-12-04T05:40:07.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f18fa74a640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f414a070 con 0x7f18f40ca990 2023-12-04T05:40:07.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7fec5d72f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fec58071080 0x7fec5806a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f21737fe640 1 -- 172.21.15.107:0/3502536844 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21680af660 con 0x7f2174159830 2023-12-04T05:40:07.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7fec5d72f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fec58071080 0x7fec5806a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.107:56652/0 (socket says 172.21.15.107:56652) 2023-12-04T05:40:07.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7fec5d72f640 1 -- 172.21.15.107:0/1775864312 learned_addr learned my addr 172.21.15.107:0/1775864312 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7fec5cf2e640 1 --2- 172.21.15.107:0/1775864312 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec58150dc0 0x7fec580c18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e4259c640 1 --2- 172.21.15.107:0/2833116655 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e440ca140 0x7f7e4406be00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7e340097a0 tx=0x7f7e3401c2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4ef7a15b8b50116 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7ff53de7c640 1 -- 172.21.15.107:0/4008331055 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff5300d8ff0 msgr2=0x7ff53004aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7ff53de7c640 1 --2- 172.21.15.107:0/4008331055 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff5300d8ff0 0x7ff53004aec0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff530036450 tx=0x7ff52c01eb90 comp rx=0 tx=0).stop 2023-12-04T05:40:07.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7ff53de7c640 1 -- 172.21.15.107:0/4008331055 shutdown_connections 2023-12-04T05:40:07.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7ff53de7c640 1 --2- 172.21.15.107:0/4008331055 >> v2:172.21.15.107:3300/0 conn(0x7ff530046780 0x7ff5300e22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7ff53de7c640 1 --2- 172.21.15.107:0/4008331055 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff5300d8ff0 0x7ff53004aec0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7ff53de7c640 1 -- 172.21.15.107:0/4008331055 >> 172.21.15.107:0/4008331055 conn(0x7ff530050160 msgr2=0x7ff530050570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e2a14d640 1 -- start start 2023-12-04T05:40:07.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e2a14d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e240c7500 0x7f7e2414efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f21737fe640 1 -- 172.21.15.107:0/3502536844 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21680af7c0 con 0x7f2174159830 2023-12-04T05:40:07.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f21737fe640 1 -- 172.21.15.107:0/3502536844 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21680afb50 con 0x7f2174159830 2023-12-04T05:40:07.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e2a14d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e24155fe0 0x7f7e24147490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e2a14d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e24159830 0x7f7e240c16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e2a14d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e24150680 con 0x7f7e240c7500 2023-12-04T05:40:07.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e2a14d640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2414eae0 con 0x7f7e24159830 2023-12-04T05:40:07.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e2a14d640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e241503e0 con 0x7f7e24155fe0 2023-12-04T05:40:07.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7ff53de7c640 1 -- 172.21.15.107:0/4008331055 shutdown_connections 2023-12-04T05:40:07.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7ff53de7c640 1 -- 172.21.15.107:0/4008331055 wait complete. 2023-12-04T05:40:07.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f415884a640 1 -- 172.21.15.107:0/823542579 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f414c04d170 msgr2=0x7f414c04ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f415884a640 1 --2- 172.21.15.107:0/823542579 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f414c04d170 0x7f414c04ef50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f41440097c0 tx=0x7f414401c4c0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f6c6f7fe640 1 --2- 172.21.15.107:0/3523059291 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6c700c7500 0x7f6c70157fb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009db0 tx=0x7f6c6800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f18ebfff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f18f40ca990 0x7f18f41f64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f18f3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18f40c9a10 0x7f18f41674f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f18f3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18f40c9a10 0x7f18f41674f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33978/0 (socket says 172.21.15.107:33978) 2023-12-04T05:40:07.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f18f3fff640 1 -- 172.21.15.107:0/2351019450 learned_addr learned my addr 172.21.15.107:0/2351019450 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e237fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e240c7500 0x7f7e2414efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e237fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e240c7500 0x7f7e2414efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33980/0 (socket says 172.21.15.107:33980) 2023-12-04T05:40:07.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e237fe640 1 -- 172.21.15.107:0/3297003789 learned_addr learned my addr 172.21.15.107:0/3297003789 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f415884a640 1 -- 172.21.15.107:0/823542579 shutdown_connections 2023-12-04T05:40:07.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f415884a640 1 --2- 172.21.15.107:0/823542579 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f414c04d170 0x7f414c04ef50 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f415884a640 1 --2- 172.21.15.107:0/823542579 >> v2:172.21.15.195:3300/0 conn(0x7f414c04e130 0x7f414c0dd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f415884a640 1 -- 172.21.15.107:0/823542579 >> 172.21.15.107:0/823542579 conn(0x7f414c00e270 msgr2=0x7f414c00e680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7fec5df30640 1 --2- 172.21.15.107:0/1775864312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec58156b70 0x7fec581fb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f18f8ce7640 1 --2- 172.21.15.107:0/2351019450 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18f4151fb0 0x7f18f4202c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7fec5d72f640 1 -- 172.21.15.107:0/1775864312 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec58150dc0 msgr2=0x7fec580c18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7fec5d72f640 1 --2- 172.21.15.107:0/1775864312 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec58150dc0 0x7fec580c18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7fec5d72f640 1 -- 172.21.15.107:0/1775864312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec58156b70 msgr2=0x7fec581fb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7fec5d72f640 1 --2- 172.21.15.107:0/1775864312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec58156b70 0x7fec581fb070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7fec5d72f640 1 -- 172.21.15.107:0/1775864312 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec50009a70 con 0x7fec58071080 2023-12-04T05:40:07.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e237fe640 1 -- 172.21.15.107:0/3297003789 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e24155fe0 msgr2=0x7f7e24147490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e237fe640 1 --2- 172.21.15.107:0/3297003789 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e24155fe0 0x7f7e24147490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e237fe640 1 -- 172.21.15.107:0/3297003789 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e24159830 msgr2=0x7f7e240c16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e237fe640 1 --2- 172.21.15.107:0/3297003789 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e24159830 0x7f7e240c16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e237fe640 1 -- 172.21.15.107:0/3297003789 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c009b30 con 0x7f7e240c7500 2023-12-04T05:40:07.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f415884a640 1 -- 172.21.15.107:0/823542579 shutdown_connections 2023-12-04T05:40:07.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7fec5cf2e640 1 --2- 172.21.15.107:0/1775864312 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec58150dc0 0x7fec580c18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T05:40:07.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e237fe640 1 --2- 172.21.15.107:0/3297003789 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e240c7500 0x7f7e2414efc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c00e930 tx=0x7f7e0c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f415884a640 1 -- 172.21.15.107:0/823542579 wait complete. 2023-12-04T05:40:07.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e41d9b640 1 -- 172.21.15.107:0/2833116655 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e340ab600 con 0x7f7e440ca140 2023-12-04T05:40:07.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f217ac16640 1 -- 172.21.15.107:0/3502536844 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2174159830 msgr2=0x7f217406df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f217ac16640 1 --2- 172.21.15.107:0/3502536844 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2174159830 0x7f217406df10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2168005a20 tx=0x7f216801f5f0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f217ac16640 1 -- 172.21.15.107:0/3502536844 shutdown_connections 2023-12-04T05:40:07.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f217ac16640 1 --2- 172.21.15.107:0/3502536844 >> v2:172.21.15.195:3300/0 conn(0x7f2174155fe0 0x7f2174156470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f217ac16640 1 --2- 172.21.15.107:0/3502536844 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2174159830 0x7f217406df10 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f217ac16640 1 -- 172.21.15.107:0/3502536844 >> 172.21.15.107:0/3502536844 conn(0x7f21740c7ff0 msgr2=0x7f21740c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f18f3fff640 1 -- 172.21.15.107:0/2351019450 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f18f40ca990 msgr2=0x7f18f41f64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f18f3fff640 1 --2- 172.21.15.107:0/2351019450 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f18f40ca990 0x7f18f41f64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f18f3fff640 1 -- 172.21.15.107:0/2351019450 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18f4151fb0 msgr2=0x7f18f4202c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f18f3fff640 1 --2- 172.21.15.107:0/2351019450 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18f4151fb0 0x7f18f4202c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f18f3fff640 1 -- 172.21.15.107:0/2351019450 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e4009420 con 0x7f18f40c9a10 2023-12-04T05:40:07.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f18f3fff640 1 --2- 172.21.15.107:0/2351019450 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18f40c9a10 0x7f18f41674f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f18e40190b0 tx=0x7f18e4019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f6c57fff640 1 -- 172.21.15.107:0/3523059291 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c68014480 con 0x7f6c700c7500 2023-12-04T05:40:07.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e41d9b640 1 -- 172.21.15.107:0/2833116655 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e340a7050 con 0x7f7e440ca140 2023-12-04T05:40:07.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e49010640 1 -- 172.21.15.107:0/2833116655 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e440ca140 msgr2=0x7f7e4406be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e41d9b640 1 -- 172.21.15.107:0/2833116655 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e340ab980 con 0x7f7e440ca140 2023-12-04T05:40:07.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f217ac16640 1 -- 172.21.15.107:0/3502536844 shutdown_connections 2023-12-04T05:40:07.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e49010640 1 --2- 172.21.15.107:0/2833116655 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e440ca140 0x7f7e4406be00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7e340097a0 tx=0x7f7e3401c2f0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f415884a640 1 Processor -- start 2023-12-04T05:40:07.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7ff53de7c640 1 Processor -- start 2023-12-04T05:40:07.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7ff53de7c640 1 -- start start 2023-12-04T05:40:07.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e49010640 1 -- 172.21.15.107:0/2833116655 shutdown_connections 2023-12-04T05:40:07.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e49010640 1 --2- 172.21.15.107:0/2833116655 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e440ca140 0x7f7e4406be00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e49010640 1 --2- 172.21.15.107:0/2833116655 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e44158420 0x7f7e4415ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e49010640 1 -- 172.21.15.107:0/2833116655 >> 172.21.15.107:0/2833116655 conn(0x7f7e4406b790 msgr2=0x7f7e440663d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f415884a640 1 -- start start 2023-12-04T05:40:07.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7ff53de7c640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff530005e70 0x7ff5300dbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7ff53de7c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff530046780 0x7ff53004ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7ff53de7c640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff5300d8ff0 0x7ff53004f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7ff53de7c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5300db9f0 con 0x7ff530046780 2023-12-04T05:40:07.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7ff53de7c640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53004a140 con 0x7ff5300d8ff0 2023-12-04T05:40:07.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7ff53de7c640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53004a400 con 0x7ff530005e70 2023-12-04T05:40:07.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e1bfff640 1 -- 172.21.15.107:0/3297003789 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c00c940 con 0x7f7e240c7500 2023-12-04T05:40:07.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f18f17fa640 1 -- 172.21.15.107:0/2351019450 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e40b6630 con 0x7f18f40c9a10 2023-12-04T05:40:07.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f18f17fa640 1 -- 172.21.15.107:0/2351019450 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e40a6050 con 0x7f18f40c9a10 2023-12-04T05:40:07.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f18fa74a640 1 -- 172.21.15.107:0/2351019450 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f40c9750 con 0x7f18f40c9a10 2023-12-04T05:40:07.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7fec5d72f640 1 --2- 172.21.15.107:0/1775864312 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fec58071080 0x7fec5806a810 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fec4800b4a0 tx=0x7fec4800b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f18fa74a640 1 -- 172.21.15.107:0/2351019450 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f41490d0 con 0x7f18f40c9a10 2023-12-04T05:40:07.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f18f17fa640 1 -- 172.21.15.107:0/2351019450 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e40b69d0 con 0x7f18f40c9a10 2023-12-04T05:40:07.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7ff537fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff530046780 0x7ff53004ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7ff537fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff530046780 0x7ff53004ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33990/0 (socket says 172.21.15.107:33990) 2023-12-04T05:40:07.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7ff537fff640 1 -- 172.21.15.107:0/23228229 learned_addr learned my addr 172.21.15.107:0/23228229 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f7e2a14d640 1 -- 172.21.15.107:0/3297003789 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e240c2f90 con 0x7f7e240c7500 2023-12-04T05:40:07.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f7e2a14d640 1 -- 172.21.15.107:0/3297003789 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2414e770 con 0x7f7e240c7500 2023-12-04T05:40:07.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f7e1bfff640 1 -- 172.21.15.107:0/3297003789 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0c00caa0 con 0x7f7e240c7500 2023-12-04T05:40:07.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f6c75d19640 1 -- 172.21.15.107:0/3523059291 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c701e1e00 con 0x7f6c700c7500 2023-12-04T05:40:07.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f6c75d19640 1 -- 172.21.15.107:0/3523059291 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c701e2590 con 0x7f6c700c7500 2023-12-04T05:40:07.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f6c57fff640 1 -- 172.21.15.107:0/3523059291 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c6800f040 con 0x7f6c700c7500 2023-12-04T05:40:07.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f6c57fff640 1 -- 172.21.15.107:0/3523059291 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c68014800 con 0x7f6c700c7500 2023-12-04T05:40:07.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f6c75d19640 1 -- 172.21.15.107:0/3523059291 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6c50000fd0 con 0x7f6c700c7500 2023-12-04T05:40:07.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7fec4dffb640 1 -- 172.21.15.107:0/1775864312 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec48010610 con 0x7fec58071080 2023-12-04T05:40:07.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7ff53d67b640 1 --2- 172.21.15.107:0/23228229 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff5300d8ff0 0x7ff53004f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7ff53ce7a640 1 --2- 172.21.15.107:0/23228229 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff530005e70 0x7ff5300dbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7ff537fff640 1 -- 172.21.15.107:0/23228229 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff530005e70 msgr2=0x7ff5300dbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7ff537fff640 1 --2- 172.21.15.107:0/23228229 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff530005e70 0x7ff5300dbe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7ff537fff640 1 -- 172.21.15.107:0/23228229 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff5300d8ff0 msgr2=0x7ff53004f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7ff537fff640 1 --2- 172.21.15.107:0/23228229 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff5300d8ff0 0x7ff53004f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7ff537fff640 1 -- 172.21.15.107:0/23228229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff52c009420 con 0x7ff530046780 2023-12-04T05:40:07.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f7e49010640 1 -- 172.21.15.107:0/2833116655 shutdown_connections 2023-12-04T05:40:07.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7ff537fff640 1 --2- 172.21.15.107:0/23228229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff530046780 0x7ff53004ef90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff52400c420 tx=0x7ff52400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f7e1bfff640 1 -- 172.21.15.107:0/3297003789 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c00ce10 con 0x7f7e240c7500 2023-12-04T05:40:07.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.351+0000 7f217ac16640 1 -- 172.21.15.107:0/3502536844 wait complete. 2023-12-04T05:40:07.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f217ac16640 1 Processor -- start 2023-12-04T05:40:07.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7fec4dffb640 1 -- 172.21.15.107:0/1775864312 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec48010790 con 0x7fec58071080 2023-12-04T05:40:07.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7fec4dffb640 1 -- 172.21.15.107:0/1775864312 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec48010b40 con 0x7fec58071080 2023-12-04T05:40:07.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7fec5f993640 1 -- 172.21.15.107:0/1775864312 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec580c44e0 con 0x7fec58071080 2023-12-04T05:40:07.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7fec5f993640 1 -- 172.21.15.107:0/1775864312 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec580c2e50 con 0x7fec58071080 2023-12-04T05:40:07.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7fec5f993640 1 -- 172.21.15.107:0/1775864312 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fec58163d10 con 0x7fec58071080 2023-12-04T05:40:07.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f18f17fa640 1 -- 172.21.15.107:0/2351019450 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f18e40b4040 con 0x7f18f40c9a10 2023-12-04T05:40:07.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f18f17fa640 1 --2- 172.21.15.107:0/2351019450 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f18d4066950 0x7f18d4068e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f415884a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f414c04d170 0x7f414c1748b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f415884a640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f414c0d7060 0x7f414c037700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f415884a640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f414c0d75d0 0x7f414c0dea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f415884a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414c0e00a0 con 0x7f414c04d170 2023-12-04T05:40:07.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f415884a640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414c0dc740 con 0x7f414c0d7060 2023-12-04T05:40:07.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f415884a640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f414c174df0 con 0x7f414c0d75d0 2023-12-04T05:40:07.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f415259c640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f414c0d7060 0x7f414c037700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f415904b640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f414c0d75d0 0x7f414c0dea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f415259c640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f414c0d7060 0x7f414c037700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.107:51530/0 (socket says 172.21.15.107:51530) 2023-12-04T05:40:07.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f415259c640 1 -- 172.21.15.107:0/2031896856 learned_addr learned my addr 172.21.15.107:0/2031896856 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f415259c640 1 -- 172.21.15.107:0/2031896856 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f414c0d75d0 msgr2=0x7f414c0dea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f415259c640 1 --2- 172.21.15.107:0/2031896856 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f414c0d75d0 0x7f414c0dea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f415259c640 1 -- 172.21.15.107:0/2031896856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f414c04d170 msgr2=0x7f414c1748b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f415259c640 1 --2- 172.21.15.107:0/2031896856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f414c04d170 0x7f414c1748b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f415259c640 1 -- 172.21.15.107:0/2031896856 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4144009420 con 0x7f414c0d7060 2023-12-04T05:40:07.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f415259c640 1 --2- 172.21.15.107:0/2031896856 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f414c0d7060 0x7f414c037700 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f415402cf30 tx=0x7f4154068290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f413b7fe640 1 -- 172.21.15.107:0/2031896856 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4154075030 con 0x7f414c0d7060 2023-12-04T05:40:07.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f415884a640 1 -- 172.21.15.107:0/2031896856 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f414c04a8c0 con 0x7f414c0d7060 2023-12-04T05:40:07.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f413b7fe640 1 -- 172.21.15.107:0/2031896856 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4154075190 con 0x7f414c0d7060 2023-12-04T05:40:07.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f413b7fe640 1 -- 172.21.15.107:0/2031896856 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415406b930 con 0x7f414c0d7060 2023-12-04T05:40:07.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f217ac16640 1 -- start start 2023-12-04T05:40:07.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f217ac16640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f21740c7500 0x7f21741fcc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f217ac16640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2174155fe0 0x7f217414f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f217ac16640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2174159830 0x7f21741520a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f217ac16640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21740ca910 con 0x7f2174155fe0 2023-12-04T05:40:07.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f217ac16640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2174151560 con 0x7f2174159830 2023-12-04T05:40:07.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f217ac16640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21740cb0e0 con 0x7f21740c7500 2023-12-04T05:40:07.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f2173fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2174155fe0 0x7f217414f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f2173fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2174155fe0 0x7f217414f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34004/0 (socket says 172.21.15.107:34004) 2023-12-04T05:40:07.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f2173fff640 1 -- 172.21.15.107:0/3980310673 learned_addr learned my addr 172.21.15.107:0/3980310673 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f21791b3640 1 --2- 172.21.15.107:0/3980310673 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2174159830 0x7f21741520a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f18f17fa640 1 -- 172.21.15.107:0/2351019450 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18e411df10 con 0x7f18f40c9a10 2023-12-04T05:40:07.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f18fa74a640 1 -- 172.21.15.107:0/2351019450 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f18d8000fd0 con 0x7f18f40c9a10 2023-12-04T05:40:07.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f7e49010640 1 -- 172.21.15.107:0/2833116655 wait complete. 2023-12-04T05:40:07.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f7e49010640 1 Processor -- start 2023-12-04T05:40:07.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f2173fff640 1 -- 172.21.15.107:0/3980310673 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f21740c7500 msgr2=0x7f21741fcc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f2173fff640 1 --2- 172.21.15.107:0/3980310673 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f21740c7500 0x7f21741fcc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f2173fff640 1 -- 172.21.15.107:0/3980310673 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2174159830 msgr2=0x7f21741520a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f2173fff640 1 --2- 172.21.15.107:0/3980310673 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2174159830 0x7f21741520a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f2173fff640 1 -- 172.21.15.107:0/3980310673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2168007cd0 con 0x7f2174155fe0 2023-12-04T05:40:07.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f18f17fa640 1 --2- 172.21.15.107:0/2351019450 >> [v2:172.21.15.107:6816/768714279,v1:172.21.15.107:6817/768714279] conn(0x7f18d406eaf0 0x7f18d4070f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f18f17fa640 1 -- 172.21.15.107:0/2351019450 --> [v2:172.21.15.107:6816/768714279,v1:172.21.15.107:6817/768714279] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18d40715c0 con 0x7f18d406eaf0 2023-12-04T05:40:07.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.355+0000 7f6c57fff640 1 -- 172.21.15.107:0/3523059291 <== mon.2 v2:172.21.15.195:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f6c68014960 con 0x7f6c700c7500 2023-12-04T05:40:07.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f6c57fff640 1 --2- 172.21.15.107:0/3523059291 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6c5c066950 0x7f6c5c068e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f6c57fff640 1 -- 172.21.15.107:0/3523059291 <== mon.2 v2:172.21.15.195:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6c68085ac0 con 0x7f6c700c7500 2023-12-04T05:40:07.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f2173fff640 1 --2- 172.21.15.107:0/3980310673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2174155fe0 0x7f217414f1e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f216000c530 tx=0x7f216000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7ff5357fa640 1 -- 172.21.15.107:0/23228229 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff524005cf0 con 0x7ff530046780 2023-12-04T05:40:07.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7ff5357fa640 1 -- 172.21.15.107:0/23228229 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff524005e50 con 0x7ff530046780 2023-12-04T05:40:07.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7ff53de7c640 1 -- 172.21.15.107:0/23228229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff530032790 con 0x7ff530046780 2023-12-04T05:40:07.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7ff5357fa640 1 -- 172.21.15.107:0/23228229 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5240072d0 con 0x7ff530046780 2023-12-04T05:40:07.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f6c57fff640 1 --2- 172.21.15.107:0/3523059291 >> [v2:172.21.15.107:6808/3097298224,v1:172.21.15.107:6809/3097298224] conn(0x7f6c5c06eb70 0x7f6c5c070f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f6c57fff640 1 -- 172.21.15.107:0/3523059291 --> [v2:172.21.15.107:6808/3097298224,v1:172.21.15.107:6809/3097298224] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c5c071640 con 0x7f6c5c06eb70 2023-12-04T05:40:07.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f6c57fff640 1 -- 172.21.15.107:0/3523059291 <== mon.2 v2:172.21.15.195:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c68085e30 con 0x7f6c700c7500 2023-12-04T05:40:07.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f18f8ce7640 1 --2- 172.21.15.107:0/2351019450 >> [v2:172.21.15.107:6816/768714279,v1:172.21.15.107:6817/768714279] conn(0x7f18d406eaf0 0x7f18d4070f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f18ebfff640 1 --2- 172.21.15.107:0/2351019450 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f18d4066950 0x7f18d4068e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7ff53de7c640 1 -- 172.21.15.107:0/23228229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5300db680 con 0x7ff530046780 2023-12-04T05:40:07.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7ff53de7c640 1 -- 172.21.15.107:0/23228229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff518000fd0 con 0x7ff530046780 2023-12-04T05:40:07.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f6c6ffff640 1 --2- 172.21.15.107:0/3523059291 >> [v2:172.21.15.107:6808/3097298224,v1:172.21.15.107:6809/3097298224] conn(0x7f6c5c06eb70 0x7f6c5c070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f6c6effd640 1 --2- 172.21.15.107:0/3523059291 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6c5c066950 0x7f6c5c068e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e1bfff640 1 -- 172.21.15.107:0/3297003789 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f7e0c013470 con 0x7f7e240c7500 2023-12-04T05:40:07.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e1bfff640 1 --2- 172.21.15.107:0/3297003789 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7e040669a0 0x7f7e04068e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e1bfff640 1 -- 172.21.15.107:0/3297003789 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e0c084620 con 0x7f7e240c7500 2023-12-04T05:40:07.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e22ffd640 1 --2- 172.21.15.107:0/3297003789 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7e040669a0 0x7f7e04068e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e2a14d640 1 --2- 172.21.15.107:0/3297003789 >> [v2:172.21.15.107:6818/3773076671,v1:172.21.15.107:6820/3773076671] conn(0x7f7e080015f0 0x7f7e08003ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e2a14d640 1 -- 172.21.15.107:0/3297003789 --> [v2:172.21.15.107:6818/3773076671,v1:172.21.15.107:6820/3773076671] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e08006b50 con 0x7f7e080015f0 2023-12-04T05:40:07.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e49010640 1 -- start start 2023-12-04T05:40:07.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e49010640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e4406c340 0x7f7e44155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e49010640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e440ca140 0x7f7e44155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e49010640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e44158420 0x7f7e441f7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e49010640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e44153530 con 0x7f7e44158420 2023-12-04T05:40:07.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e49010640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e440cefb0 con 0x7f7e4406c340 2023-12-04T05:40:07.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e49010640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e440ce9f0 con 0x7f7e440ca140 2023-12-04T05:40:07.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e4359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e44158420 0x7f7e441f7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e4359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e44158420 0x7f7e441f7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34018/0 (socket says 172.21.15.107:34018) 2023-12-04T05:40:07.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e4359e640 1 -- 172.21.15.107:0/727815042 learned_addr learned my addr 172.21.15.107:0/727815042 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f415884a640 1 -- 172.21.15.107:0/2031896856 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f414c0dfdc0 con 0x7f414c0d7060 2023-12-04T05:40:07.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f415884a640 1 -- 172.21.15.107:0/2031896856 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f414c047850 con 0x7f414c0d7060 2023-12-04T05:40:07.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e4259c640 1 --2- 172.21.15.107:0/727815042 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e440ca140 0x7f7e44155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f18ebfff640 1 --2- 172.21.15.107:0/2351019450 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f18d4066950 0x7f18d4068e10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f18f4167a30 tx=0x7f18dc009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f6c6effd640 1 --2- 172.21.15.107:0/3523059291 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6c5c066950 0x7f6c5c068e10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6c701f12d0 tx=0x7f6c60009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e23fff640 1 --2- 172.21.15.107:0/3297003789 >> [v2:172.21.15.107:6818/3773076671,v1:172.21.15.107:6820/3773076671] conn(0x7f7e080015f0 0x7f7e08003ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f6c6ffff640 1 --2- 172.21.15.107:0/3523059291 >> [v2:172.21.15.107:6808/3097298224,v1:172.21.15.107:6809/3097298224] conn(0x7f6c5c06eb70 0x7f6c5c070f90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e4359e640 1 -- 172.21.15.107:0/727815042 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e440ca140 msgr2=0x7f7e44155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e4359e640 1 --2- 172.21.15.107:0/727815042 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e440ca140 0x7f7e44155a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e4359e640 1 -- 172.21.15.107:0/727815042 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e4406c340 msgr2=0x7f7e44155520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e4359e640 1 --2- 172.21.15.107:0/727815042 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e4406c340 0x7f7e44155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e4359e640 1 -- 172.21.15.107:0/727815042 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e340093b0 con 0x7f7e44158420 2023-12-04T05:40:07.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e22ffd640 1 --2- 172.21.15.107:0/3297003789 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7e040669a0 0x7f7e04068e60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7e14000f70 tx=0x7f7e140079e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f21717fa640 1 -- 172.21.15.107:0/3980310673 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2160005cf0 con 0x7f2174155fe0 2023-12-04T05:40:07.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f21717fa640 1 -- 172.21.15.107:0/3980310673 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2160005e50 con 0x7f2174155fe0 2023-12-04T05:40:07.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f21717fa640 1 -- 172.21.15.107:0/3980310673 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2160007270 con 0x7f2174155fe0 2023-12-04T05:40:07.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f18f17fa640 1 -- 172.21.15.107:0/2351019450 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18e40eb230 con 0x7f18f40c9a10 2023-12-04T05:40:07.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e23fff640 1 --2- 172.21.15.107:0/3297003789 >> [v2:172.21.15.107:6818/3773076671,v1:172.21.15.107:6820/3773076671] conn(0x7f7e080015f0 0x7f7e08003ab0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e4359e640 1 --2- 172.21.15.107:0/727815042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e44158420 0x7f7e441f7890 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c00e780 tx=0x7f7e3c00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7fec4dffb640 1 -- 172.21.15.107:0/1775864312 <== mon.2 v2:172.21.15.195:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fec48010ca0 con 0x7fec58071080 2023-12-04T05:40:07.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7fec4dffb640 1 --2- 172.21.15.107:0/1775864312 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fec3c0668d0 0x7fec3c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7fec4dffb640 1 -- 172.21.15.107:0/1775864312 <== mon.2 v2:172.21.15.195:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec480846c0 con 0x7fec58071080 2023-12-04T05:40:07.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7fec5cf2e640 1 --2- 172.21.15.107:0/1775864312 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fec3c0668d0 0x7fec3c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7fec4dffb640 1 --2- 172.21.15.107:0/1775864312 >> [v2:172.21.15.195:6808/768189614,v1:172.21.15.195:6809/768189614] conn(0x7fec3c06ea70 0x7fec3c070e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7fec4dffb640 1 -- 172.21.15.107:0/1775864312 --> [v2:172.21.15.195:6808/768189614,v1:172.21.15.195:6809/768189614] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec3c071540 con 0x7fec3c06ea70 2023-12-04T05:40:07.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7fec4dffb640 1 -- 172.21.15.107:0/1775864312 <== mon.2 v2:172.21.15.195:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec48084b10 con 0x7fec58071080 2023-12-04T05:40:07.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f413b7fe640 1 -- 172.21.15.107:0/2031896856 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f415406ba90 con 0x7f414c0d7060 2023-12-04T05:40:07.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e2b7fe640 1 -- 172.21.15.107:0/727815042 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c00f930 con 0x7f7e44158420 2023-12-04T05:40:07.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e2b7fe640 1 -- 172.21.15.107:0/727815042 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3c00fa90 con 0x7f7e44158420 2023-12-04T05:40:07.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e49010640 1 -- 172.21.15.107:0/727815042 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e440cecd0 con 0x7f7e44158420 2023-12-04T05:40:07.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7fec5cf2e640 1 --2- 172.21.15.107:0/1775864312 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fec3c0668d0 0x7fec3c068d90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fec58144e20 tx=0x7fec54007e90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f18f8ce7640 1 --2- 172.21.15.107:0/2351019450 >> [v2:172.21.15.107:6816/768714279,v1:172.21.15.107:6817/768714279] conn(0x7f18d406eaf0 0x7f18d4070f10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e49010640 1 -- 172.21.15.107:0/727815042 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e44154d20 con 0x7f7e44158420 2023-12-04T05:40:07.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f7e1bfff640 1 -- 172.21.15.107:0/3297003789 <== osd.3 v2:172.21.15.107:6818/3773076671 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f7e08006b50 con 0x7f7e080015f0 2023-12-04T05:40:07.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f217ac16640 1 -- 172.21.15.107:0/3980310673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21741fd1d0 con 0x7f2174155fe0 2023-12-04T05:40:07.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f217ac16640 1 -- 172.21.15.107:0/3980310673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2174151280 con 0x7f2174155fe0 2023-12-04T05:40:07.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7f217ac16640 1 -- 172.21.15.107:0/3980310673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2154000fd0 con 0x7f2174155fe0 2023-12-04T05:40:07.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7ff5357fa640 1 -- 172.21.15.107:0/23228229 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff524007430 con 0x7ff530046780 2023-12-04T05:40:07.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7f7e2b7fe640 1 -- 172.21.15.107:0/727815042 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c00fe60 con 0x7f7e44158420 2023-12-04T05:40:07.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f6c57fff640 1 -- 172.21.15.107:0/3523059291 <== osd.2 v2:172.21.15.107:6808/3097298224 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f6c5c071640 con 0x7f6c5c06eb70 2023-12-04T05:40:07.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f413b7fe640 1 --2- 172.21.15.107:0/2031896856 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f41400668d0 0x7f4140068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.359+0000 7f413b7fe640 1 -- 172.21.15.107:0/2031896856 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4154110850 con 0x7f414c0d7060 2023-12-04T05:40:07.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7f413b7fe640 1 --2- 172.21.15.107:0/2031896856 >> [v2:172.21.15.195:6824/1484264540,v1:172.21.15.195:6825/1484264540] conn(0x7f414006ea70 0x7f4140070e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7f413b7fe640 1 -- 172.21.15.107:0/2031896856 --> [v2:172.21.15.195:6824/1484264540,v1:172.21.15.195:6825/1484264540] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4140071540 con 0x7f414006ea70 2023-12-04T05:40:07.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7f413b7fe640 1 -- 172.21.15.107:0/2031896856 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4154110bc0 con 0x7f414c0d7060 2023-12-04T05:40:07.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7fec5df30640 1 --2- 172.21.15.107:0/1775864312 >> [v2:172.21.15.195:6808/768189614,v1:172.21.15.195:6809/768189614] conn(0x7fec3c06ea70 0x7fec3c070e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7ff5357fa640 1 --2- 172.21.15.107:0/23228229 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff51c066920 0x7ff51c068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7f18f17fa640 1 -- 172.21.15.107:0/2351019450 <== osd.0 v2:172.21.15.107:6816/768714279 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f18d40715c0 con 0x7f18d406eaf0 2023-12-04T05:40:07.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7ff5357fa640 1 -- 172.21.15.107:0/23228229 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff524083e20 con 0x7ff530046780 2023-12-04T05:40:07.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7ff53ce7a640 1 --2- 172.21.15.107:0/23228229 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff51c066920 0x7ff51c068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7ff5357fa640 1 --2- 172.21.15.107:0/23228229 >> [v2:172.21.15.107:6800/2496355981,v1:172.21.15.107:6801/2496355981] conn(0x7ff51c06eb40 0x7ff51c070f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7ff5357fa640 1 -- 172.21.15.107:0/23228229 --> [v2:172.21.15.107:6800/2496355981,v1:172.21.15.107:6801/2496355981] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff51c071610 con 0x7ff51c06eb40 2023-12-04T05:40:07.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7ff5357fa640 1 -- 172.21.15.107:0/23228229 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5240842c0 con 0x7ff530046780 2023-12-04T05:40:07.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7ff53d67b640 1 --2- 172.21.15.107:0/23228229 >> [v2:172.21.15.107:6800/2496355981,v1:172.21.15.107:6801/2496355981] conn(0x7ff51c06eb40 0x7ff51c070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7ff53d67b640 1 --2- 172.21.15.107:0/23228229 >> [v2:172.21.15.107:6800/2496355981,v1:172.21.15.107:6801/2496355981] conn(0x7ff51c06eb40 0x7ff51c070f60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7f7e49010640 1 -- 172.21.15.107:0/727815042 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7e24000fd0 con 0x7f7e44158420 2023-12-04T05:40:07.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7ff5357fa640 1 -- 172.21.15.107:0/23228229 <== osd.1 v2:172.21.15.107:6800/2496355981 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7ff51c071610 con 0x7ff51c06eb40 2023-12-04T05:40:07.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7ff53ce7a640 1 --2- 172.21.15.107:0/23228229 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff51c066920 0x7ff51c068de0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff52c0b2000 tx=0x7ff52c0aa070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7f21717fa640 1 -- 172.21.15.107:0/3980310673 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f21600073d0 con 0x7f2174155fe0 2023-12-04T05:40:07.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7f21717fa640 1 --2- 172.21.15.107:0/3980310673 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2158066920 0x7f2158068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7f7e2b7fe640 1 -- 172.21.15.107:0/727815042 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f7e3c015070 con 0x7f7e44158420 2023-12-04T05:40:07.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7f21717fa640 1 -- 172.21.15.107:0/3980310673 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2160084130 con 0x7f2174155fe0 2023-12-04T05:40:07.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7f21789b2640 1 --2- 172.21.15.107:0/3980310673 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2158066920 0x7f2158068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7f21717fa640 1 --2- 172.21.15.107:0/3980310673 >> [v2:172.21.15.195:6811/2638471412,v1:172.21.15.195:6813/2638471412] conn(0x7f215806eb40 0x7f2158070f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7f21717fa640 1 -- 172.21.15.107:0/3980310673 --> [v2:172.21.15.195:6811/2638471412,v1:172.21.15.195:6813/2638471412] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2158071610 con 0x7f215806eb40 2023-12-04T05:40:07.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7f21717fa640 1 -- 172.21.15.107:0/3980310673 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21600844a0 con 0x7f2174155fe0 2023-12-04T05:40:07.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7f7e2b7fe640 1 --2- 172.21.15.107:0/727815042 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7e30066920 0x7f7e30068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.363+0000 7f21791b3640 1 --2- 172.21.15.107:0/3980310673 >> [v2:172.21.15.195:6811/2638471412,v1:172.21.15.195:6813/2638471412] conn(0x7f215806eb40 0x7f2158070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.367+0000 7f7e2b7fe640 1 -- 172.21.15.107:0/727815042 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e3c087590 con 0x7f7e44158420 2023-12-04T05:40:07.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.367+0000 7f21789b2640 1 --2- 172.21.15.107:0/3980310673 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2158066920 0x7f2158068de0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f21680b3000 tx=0x7f2168004740 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.367+0000 7f7e2b7fe640 1 --2- 172.21.15.107:0/727815042 >> [v2:172.21.15.195:6800/2747823542,v1:172.21.15.195:6801/2747823542] conn(0x7f7e3006eb40 0x7f7e30070f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.367+0000 7f7e2b7fe640 1 -- 172.21.15.107:0/727815042 --> [v2:172.21.15.195:6800/2747823542,v1:172.21.15.195:6801/2747823542] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e30071610 con 0x7f7e3006eb40 2023-12-04T05:40:07.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.367+0000 7f7e2b7fe640 1 -- 172.21.15.107:0/727815042 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e3c087900 con 0x7f7e44158420 2023-12-04T05:40:07.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.367+0000 7f7e42d9d640 1 --2- 172.21.15.107:0/727815042 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7e30066920 0x7f7e30068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.367+0000 7f4152d9d640 1 --2- 172.21.15.107:0/2031896856 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f41400668d0 0x7f4140068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.367+0000 7f7e42d9d640 1 --2- 172.21.15.107:0/727815042 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7e30066920 0x7f7e30068de0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009690 tx=0x7f7e38006c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.367+0000 7f21791b3640 1 --2- 172.21.15.107:0/3980310673 >> [v2:172.21.15.195:6811/2638471412,v1:172.21.15.195:6813/2638471412] conn(0x7f215806eb40 0x7f2158070f60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.367+0000 7f415904b640 1 --2- 172.21.15.107:0/2031896856 >> [v2:172.21.15.195:6824/1484264540,v1:172.21.15.195:6825/1484264540] conn(0x7f414006ea70 0x7f4140070e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.367+0000 7f4152d9d640 1 --2- 172.21.15.107:0/2031896856 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f41400668d0 0x7f4140068d90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4148001560 tx=0x7f4148007a50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.367+0000 7f7e4259c640 1 --2- 172.21.15.107:0/727815042 >> [v2:172.21.15.195:6800/2747823542,v1:172.21.15.195:6801/2747823542] conn(0x7f7e3006eb40 0x7f7e30070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.367+0000 7f415904b640 1 --2- 172.21.15.107:0/2031896856 >> [v2:172.21.15.195:6824/1484264540,v1:172.21.15.195:6825/1484264540] conn(0x7f414006ea70 0x7f4140070e90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.367+0000 7fec5df30640 1 --2- 172.21.15.107:0/1775864312 >> [v2:172.21.15.195:6808/768189614,v1:172.21.15.195:6809/768189614] conn(0x7fec3c06ea70 0x7fec3c070e90 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.367+0000 7f7e4259c640 1 --2- 172.21.15.107:0/727815042 >> [v2:172.21.15.195:6800/2747823542,v1:172.21.15.195:6801/2747823542] conn(0x7f7e3006eb40 0x7f7e30070f60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.367+0000 7f21717fa640 1 -- 172.21.15.107:0/3980310673 <== osd.6 v2:172.21.15.195:6811/2638471412 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f2158071610 con 0x7f215806eb40 2023-12-04T05:40:07.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.367+0000 7fec4dffb640 1 -- 172.21.15.107:0/1775864312 <== osd.7 v2:172.21.15.195:6808/768189614 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fec3c071540 con 0x7fec3c06ea70 2023-12-04T05:40:07.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.371+0000 7f7e2b7fe640 1 -- 172.21.15.107:0/727815042 <== osd.4 v2:172.21.15.195:6800/2747823542 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f7e30071610 con 0x7f7e3006eb40 2023-12-04T05:40:07.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.379+0000 7f413b7fe640 1 -- 172.21.15.107:0/2031896856 <== osd.5 v2:172.21.15.195:6824/1484264540 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f4140071540 con 0x7f414006ea70 2023-12-04T05:40:07.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.383+0000 7f7e49010640 1 -- 172.21.15.107:0/727815042 --> [v2:172.21.15.195:6800/2747823542,v1:172.21.15.195:6801/2747823542] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7e24002cf0 con 0x7f7e3006eb40 2023-12-04T05:40:07.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.387+0000 7f6c75d19640 1 -- 172.21.15.107:0/3523059291 --> [v2:172.21.15.107:6808/3097298224,v1:172.21.15.107:6809/3097298224] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6c50002cf0 con 0x7f6c5c06eb70 2023-12-04T05:40:07.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.387+0000 7f6c57fff640 1 -- 172.21.15.107:0/3523059291 <== osd.2 v2:172.21.15.107:6808/3097298224 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6c50002cf0 con 0x7f6c5c06eb70 2023-12-04T05:40:07.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.387+0000 7f7e2b7fe640 1 -- 172.21.15.107:0/727815042 <== osd.4 v2:172.21.15.195:6800/2747823542 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7e24002cf0 con 0x7f7e3006eb40 2023-12-04T05:40:07.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.387+0000 7f7e49010640 1 -- 172.21.15.107:0/727815042 >> [v2:172.21.15.195:6800/2747823542,v1:172.21.15.195:6801/2747823542] conn(0x7f7e3006eb40 msgr2=0x7f7e30070f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.387+0000 7f7e49010640 1 --2- 172.21.15.107:0/727815042 >> [v2:172.21.15.195:6800/2747823542,v1:172.21.15.195:6801/2747823542] conn(0x7f7e3006eb40 0x7f7e30070f60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.387+0000 7f7e49010640 1 -- 172.21.15.107:0/727815042 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7e30066920 msgr2=0x7f7e30068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.387+0000 7f7e49010640 1 --2- 172.21.15.107:0/727815042 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7e30066920 0x7f7e30068de0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009690 tx=0x7f7e38006c00 comp rx=0 tx=0).stop 2023-12-04T05:40:07.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.387+0000 7f7e49010640 1 -- 172.21.15.107:0/727815042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e44158420 msgr2=0x7f7e441f7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.387+0000 7f7e49010640 1 --2- 172.21.15.107:0/727815042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e44158420 0x7f7e441f7890 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c00e780 tx=0x7f7e3c00ec40 comp rx=0 tx=0).stop 2023-12-04T05:40:07.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.387+0000 7f7e4359e640 1 -- 172.21.15.107:0/727815042 reap_dead start 2023-12-04T05:40:07.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.387+0000 7f7e49010640 1 -- 172.21.15.107:0/727815042 shutdown_connections 2023-12-04T05:40:07.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.387+0000 7f7e49010640 1 -- 172.21.15.107:0/727815042 >> 172.21.15.107:0/727815042 conn(0x7f7e4406b790 msgr2=0x7f7e44156ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.387+0000 7f7e49010640 1 -- 172.21.15.107:0/727815042 shutdown_connections 2023-12-04T05:40:07.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.387+0000 7f7e49010640 1 -- 172.21.15.107:0/727815042 wait complete. 2023-12-04T05:40:07.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7ff53de7c640 1 -- 172.21.15.107:0/23228229 --> [v2:172.21.15.107:6800/2496355981,v1:172.21.15.107:6801/2496355981] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff518002cf0 con 0x7ff51c06eb40 2023-12-04T05:40:07.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f18fa74a640 1 -- 172.21.15.107:0/2351019450 --> [v2:172.21.15.107:6816/768714279,v1:172.21.15.107:6817/768714279] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f18d8002cc0 con 0x7f18d406eaf0 2023-12-04T05:40:07.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f18f17fa640 1 -- 172.21.15.107:0/2351019450 <== osd.0 v2:172.21.15.107:6816/768714279 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f18d8002cc0 con 0x7f18d406eaf0 2023-12-04T05:40:07.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7ff5357fa640 1 -- 172.21.15.107:0/23228229 <== osd.1 v2:172.21.15.107:6800/2496355981 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff518002cf0 con 0x7ff51c06eb40 2023-12-04T05:40:07.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7ff53de7c640 1 -- 172.21.15.107:0/23228229 >> [v2:172.21.15.107:6800/2496355981,v1:172.21.15.107:6801/2496355981] conn(0x7ff51c06eb40 msgr2=0x7ff51c070f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7ff53de7c640 1 --2- 172.21.15.107:0/23228229 >> [v2:172.21.15.107:6800/2496355981,v1:172.21.15.107:6801/2496355981] conn(0x7ff51c06eb40 0x7ff51c070f60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7ff53de7c640 1 -- 172.21.15.107:0/23228229 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff51c066920 msgr2=0x7ff51c068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7ff53de7c640 1 --2- 172.21.15.107:0/23228229 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff51c066920 0x7ff51c068de0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff52c0b2000 tx=0x7ff52c0aa070 comp rx=0 tx=0).stop 2023-12-04T05:40:07.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7ff53de7c640 1 -- 172.21.15.107:0/23228229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff530046780 msgr2=0x7ff53004ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7ff53de7c640 1 --2- 172.21.15.107:0/23228229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff530046780 0x7ff53004ef90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff52400c420 tx=0x7ff52400c8e0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7ff53d67b640 1 -- 172.21.15.107:0/23228229 reap_dead start 2023-12-04T05:40:07.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7ff53de7c640 1 -- 172.21.15.107:0/23228229 shutdown_connections 2023-12-04T05:40:07.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7ff53de7c640 1 -- 172.21.15.107:0/23228229 >> 172.21.15.107:0/23228229 conn(0x7ff530050160 msgr2=0x7ff5300d5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f18fa74a640 1 -- 172.21.15.107:0/2351019450 >> [v2:172.21.15.107:6816/768714279,v1:172.21.15.107:6817/768714279] conn(0x7f18d406eaf0 msgr2=0x7f18d4070f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f18fa74a640 1 --2- 172.21.15.107:0/2351019450 >> [v2:172.21.15.107:6816/768714279,v1:172.21.15.107:6817/768714279] conn(0x7f18d406eaf0 0x7f18d4070f10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7ff53de7c640 1 -- 172.21.15.107:0/23228229 shutdown_connections 2023-12-04T05:40:07.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7ff53de7c640 1 -- 172.21.15.107:0/23228229 wait complete. 2023-12-04T05:40:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f6c55ffb640 1 -- 172.21.15.107:0/3523059291 >> [v2:172.21.15.107:6808/3097298224,v1:172.21.15.107:6809/3097298224] conn(0x7f6c5c06eb70 msgr2=0x7f6c5c070f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f6c55ffb640 1 --2- 172.21.15.107:0/3523059291 >> [v2:172.21.15.107:6808/3097298224,v1:172.21.15.107:6809/3097298224] conn(0x7f6c5c06eb70 0x7f6c5c070f90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f6c55ffb640 1 -- 172.21.15.107:0/3523059291 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6c5c066950 msgr2=0x7f6c5c068e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f6c55ffb640 1 --2- 172.21.15.107:0/3523059291 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6c5c066950 0x7f6c5c068e10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6c701f12d0 tx=0x7f6c60009190 comp rx=0 tx=0).stop 2023-12-04T05:40:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f6c55ffb640 1 -- 172.21.15.107:0/3523059291 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6c700c7500 msgr2=0x7f6c70157fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f7e2a14d640 1 -- 172.21.15.107:0/3297003789 --> [v2:172.21.15.107:6818/3773076671,v1:172.21.15.107:6820/3773076671] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7e08005bd0 con 0x7f7e080015f0 2023-12-04T05:40:07.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f18fa74a640 1 -- 172.21.15.107:0/2351019450 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f18d4066950 msgr2=0x7f18d4068e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f18fa74a640 1 --2- 172.21.15.107:0/2351019450 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f18d4066950 0x7f18d4068e10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f18f4167a30 tx=0x7f18dc009190 comp rx=0 tx=0).stop 2023-12-04T05:40:07.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f18fa74a640 1 -- 172.21.15.107:0/2351019450 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18f40c9a10 msgr2=0x7f18f41674f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f6c55ffb640 1 --2- 172.21.15.107:0/3523059291 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6c700c7500 0x7f6c70157fb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009db0 tx=0x7f6c6800c490 comp rx=0 tx=0).stop 2023-12-04T05:40:07.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f18fa74a640 1 --2- 172.21.15.107:0/2351019450 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18f40c9a10 0x7f18f41674f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f18e40190b0 tx=0x7f18e4019240 comp rx=0 tx=0).stop 2023-12-04T05:40:07.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f18f8ce7640 1 -- 172.21.15.107:0/2351019450 reap_dead start 2023-12-04T05:40:07.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f6c6ffff640 1 -- 172.21.15.107:0/3523059291 reap_dead start 2023-12-04T05:40:07.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f6c55ffb640 1 -- 172.21.15.107:0/3523059291 shutdown_connections 2023-12-04T05:40:07.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f6c55ffb640 1 -- 172.21.15.107:0/3523059291 >> 172.21.15.107:0/3523059291 conn(0x7f6c700c7ff0 msgr2=0x7f6c700c9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f18fa74a640 1 -- 172.21.15.107:0/2351019450 shutdown_connections 2023-12-04T05:40:07.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f18fa74a640 1 -- 172.21.15.107:0/2351019450 >> 172.21.15.107:0/2351019450 conn(0x7f18f4066570 msgr2=0x7f18f4066950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f18fa74a640 1 -- 172.21.15.107:0/2351019450 shutdown_connections 2023-12-04T05:40:07.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f18fa74a640 1 -- 172.21.15.107:0/2351019450 wait complete. 2023-12-04T05:40:07.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f7e1bfff640 1 -- 172.21.15.107:0/3297003789 <== osd.3 v2:172.21.15.107:6818/3773076671 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7e08005bd0 con 0x7f7e080015f0 2023-12-04T05:40:07.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.391+0000 7f6c55ffb640 1 -- 172.21.15.107:0/3523059291 shutdown_connections 2023-12-04T05:40:07.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.395+0000 7f6c55ffb640 1 -- 172.21.15.107:0/3523059291 wait complete. 2023-12-04T05:40:07.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.395+0000 7f7e2a14d640 1 -- 172.21.15.107:0/3297003789 >> [v2:172.21.15.107:6818/3773076671,v1:172.21.15.107:6820/3773076671] conn(0x7f7e080015f0 msgr2=0x7f7e08003ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.395+0000 7f7e2a14d640 1 --2- 172.21.15.107:0/3297003789 >> [v2:172.21.15.107:6818/3773076671,v1:172.21.15.107:6820/3773076671] conn(0x7f7e080015f0 0x7f7e08003ab0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.395+0000 7f7e2a14d640 1 -- 172.21.15.107:0/3297003789 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7e040669a0 msgr2=0x7f7e04068e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.395+0000 7f7e2a14d640 1 --2- 172.21.15.107:0/3297003789 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7e040669a0 0x7f7e04068e60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7e14000f70 tx=0x7f7e140079e0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.395+0000 7f7e2a14d640 1 -- 172.21.15.107:0/3297003789 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e240c7500 msgr2=0x7f7e2414efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.395+0000 7f7e2a14d640 1 --2- 172.21.15.107:0/3297003789 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e240c7500 0x7f7e2414efc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c00e930 tx=0x7f7e0c00edf0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.395+0000 7f7e23fff640 1 -- 172.21.15.107:0/3297003789 reap_dead start 2023-12-04T05:40:07.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.395+0000 7f7e2a14d640 1 -- 172.21.15.107:0/3297003789 shutdown_connections 2023-12-04T05:40:07.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.395+0000 7f7e2a14d640 1 -- 172.21.15.107:0/3297003789 >> 172.21.15.107:0/3297003789 conn(0x7f7e240c7ff0 msgr2=0x7f7e24070380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.395+0000 7f7e2a14d640 1 -- 172.21.15.107:0/3297003789 shutdown_connections 2023-12-04T05:40:07.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.395+0000 7f7e2a14d640 1 -- 172.21.15.107:0/3297003789 wait complete. 2023-12-04T05:40:07.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.395+0000 7fec5f993640 1 -- 172.21.15.107:0/1775864312 --> [v2:172.21.15.195:6808/768189614,v1:172.21.15.195:6809/768189614] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fec58067ec0 con 0x7fec3c06ea70 2023-12-04T05:40:07.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.395+0000 7fec4dffb640 1 -- 172.21.15.107:0/1775864312 <== osd.7 v2:172.21.15.195:6808/768189614 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fec58067ec0 con 0x7fec3c06ea70 2023-12-04T05:40:07.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.395+0000 7fec5f993640 1 -- 172.21.15.107:0/1775864312 >> [v2:172.21.15.195:6808/768189614,v1:172.21.15.195:6809/768189614] conn(0x7fec3c06ea70 msgr2=0x7fec3c070e90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.395+0000 7fec5f993640 1 --2- 172.21.15.107:0/1775864312 >> [v2:172.21.15.195:6808/768189614,v1:172.21.15.195:6809/768189614] conn(0x7fec3c06ea70 0x7fec3c070e90 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.395+0000 7fec5f993640 1 -- 172.21.15.107:0/1775864312 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fec3c0668d0 msgr2=0x7fec3c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.395+0000 7fec5f993640 1 --2- 172.21.15.107:0/1775864312 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fec3c0668d0 0x7fec3c068d90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fec58144e20 tx=0x7fec54007e90 comp rx=0 tx=0).stop 2023-12-04T05:40:07.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.395+0000 7fec5f993640 1 -- 172.21.15.107:0/1775864312 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fec58071080 msgr2=0x7fec5806a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.395+0000 7fec5f993640 1 --2- 172.21.15.107:0/1775864312 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fec58071080 0x7fec5806a810 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fec4800b4a0 tx=0x7fec4800b960 comp rx=0 tx=0).stop 2023-12-04T05:40:07.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.395+0000 7fec5df30640 1 -- 172.21.15.107:0/1775864312 reap_dead start 2023-12-04T05:40:07.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.395+0000 7f217ac16640 1 -- 172.21.15.107:0/3980310673 --> [v2:172.21.15.195:6811/2638471412,v1:172.21.15.195:6813/2638471412] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2154002cc0 con 0x7f215806eb40 2023-12-04T05:40:07.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.399+0000 7fec5f993640 1 -- 172.21.15.107:0/1775864312 shutdown_connections 2023-12-04T05:40:07.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.399+0000 7fec5f993640 1 -- 172.21.15.107:0/1775864312 >> 172.21.15.107:0/1775864312 conn(0x7fec580cc490 msgr2=0x7fec580c7b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.399+0000 7f21717fa640 1 -- 172.21.15.107:0/3980310673 <== osd.6 v2:172.21.15.195:6811/2638471412 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2154002cc0 con 0x7f215806eb40 2023-12-04T05:40:07.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.403+0000 7fec5f993640 1 -- 172.21.15.107:0/1775864312 shutdown_connections 2023-12-04T05:40:07.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.403+0000 7fec5f993640 1 -- 172.21.15.107:0/1775864312 wait complete. 2023-12-04T05:40:07.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.403+0000 7f217ac16640 1 -- 172.21.15.107:0/3980310673 >> [v2:172.21.15.195:6811/2638471412,v1:172.21.15.195:6813/2638471412] conn(0x7f215806eb40 msgr2=0x7f2158070f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.403+0000 7f217ac16640 1 --2- 172.21.15.107:0/3980310673 >> [v2:172.21.15.195:6811/2638471412,v1:172.21.15.195:6813/2638471412] conn(0x7f215806eb40 0x7f2158070f60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.403+0000 7f217ac16640 1 -- 172.21.15.107:0/3980310673 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2158066920 msgr2=0x7f2158068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.403+0000 7f217ac16640 1 --2- 172.21.15.107:0/3980310673 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2158066920 0x7f2158068de0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f21680b3000 tx=0x7f2168004740 comp rx=0 tx=0).stop 2023-12-04T05:40:07.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.403+0000 7f217ac16640 1 -- 172.21.15.107:0/3980310673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2174155fe0 msgr2=0x7f217414f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.403+0000 7f217ac16640 1 --2- 172.21.15.107:0/3980310673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2174155fe0 0x7f217414f1e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f216000c530 tx=0x7f216000c9f0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.403+0000 7f21791b3640 1 -- 172.21.15.107:0/3980310673 reap_dead start 2023-12-04T05:40:07.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.403+0000 7f217ac16640 1 -- 172.21.15.107:0/3980310673 shutdown_connections 2023-12-04T05:40:07.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.403+0000 7f217ac16640 1 -- 172.21.15.107:0/3980310673 >> 172.21.15.107:0/3980310673 conn(0x7f21740c7ff0 msgr2=0x7f21740c2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.403+0000 7f217ac16640 1 -- 172.21.15.107:0/3980310673 shutdown_connections 2023-12-04T05:40:07.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.403+0000 7f217ac16640 1 -- 172.21.15.107:0/3980310673 wait complete. 2023-12-04T05:40:07.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.407+0000 7f415884a640 1 -- 172.21.15.107:0/2031896856 --> [v2:172.21.15.195:6824/1484264540,v1:172.21.15.195:6825/1484264540] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f414c0e02b0 con 0x7f414006ea70 2023-12-04T05:40:07.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.407+0000 7f413b7fe640 1 -- 172.21.15.107:0/2031896856 <== osd.5 v2:172.21.15.195:6824/1484264540 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f41440ab660 con 0x7f414006ea70 2023-12-04T05:40:07.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.407+0000 7f415884a640 1 -- 172.21.15.107:0/2031896856 >> [v2:172.21.15.195:6824/1484264540,v1:172.21.15.195:6825/1484264540] conn(0x7f414006ea70 msgr2=0x7f4140070e90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.407+0000 7f415884a640 1 --2- 172.21.15.107:0/2031896856 >> [v2:172.21.15.195:6824/1484264540,v1:172.21.15.195:6825/1484264540] conn(0x7f414006ea70 0x7f4140070e90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.411+0000 7f415884a640 1 -- 172.21.15.107:0/2031896856 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f41400668d0 msgr2=0x7f4140068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.411+0000 7f415884a640 1 --2- 172.21.15.107:0/2031896856 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f41400668d0 0x7f4140068d90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4148001560 tx=0x7f4148007a50 comp rx=0 tx=0).stop 2023-12-04T05:40:07.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.411+0000 7f415884a640 1 -- 172.21.15.107:0/2031896856 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f414c0d7060 msgr2=0x7f414c037700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.411+0000 7f415884a640 1 --2- 172.21.15.107:0/2031896856 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f414c0d7060 0x7f414c037700 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f415402cf30 tx=0x7f4154068290 comp rx=0 tx=0).stop 2023-12-04T05:40:07.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.411+0000 7f415904b640 1 -- 172.21.15.107:0/2031896856 reap_dead start 2023-12-04T05:40:07.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.411+0000 7f415884a640 1 -- 172.21.15.107:0/2031896856 shutdown_connections 2023-12-04T05:40:07.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.411+0000 7f415884a640 1 -- 172.21.15.107:0/2031896856 >> 172.21.15.107:0/2031896856 conn(0x7f414c00e270 msgr2=0x7f414c050d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.411+0000 7f415884a640 1 -- 172.21.15.107:0/2031896856 shutdown_connections 2023-12-04T05:40:07.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.411+0000 7f415884a640 1 -- 172.21.15.107:0/2031896856 wait complete. 2023-12-04T05:40:07.477 INFO:teuthology.orchestra.run.smithi107.stdout:55834574855 2023-12-04T05:40:07.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-04T05:40:07.478 INFO:teuthology.orchestra.run.smithi107.stdout:55834574856 2023-12-04T05:40:07.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-04T05:40:07.479 INFO:teuthology.orchestra.run.smithi107.stdout:55834574854 2023-12-04T05:40:07.480 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-04T05:40:07.481 INFO:teuthology.orchestra.run.smithi107.stdout:55834574854 2023-12-04T05:40:07.481 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-04T05:40:07.481 INFO:teuthology.orchestra.run.smithi107.stdout:55834574854 2023-12-04T05:40:07.482 INFO:teuthology.orchestra.run.smithi107.stdout:55834574856 2023-12-04T05:40:07.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-04T05:40:07.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-04T05:40:07.484 INFO:teuthology.orchestra.run.smithi107.stdout:55834574854 2023-12-04T05:40:07.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-04T05:40:07.485 INFO:teuthology.orchestra.run.smithi107.stdout:55834574856 2023-12-04T05:40:07.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-04T05:40:07.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.687+0000 7efe0ffff640 1 Processor -- start 2023-12-04T05:40:07.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.687+0000 7efe0ffff640 1 -- start start 2023-12-04T05:40:07.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ffb6fd54640 1 Processor -- start 2023-12-04T05:40:07.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ffb6fd54640 1 -- start start 2023-12-04T05:40:07.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7efe0ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efe1014c6c0 0x7efe100c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7efe0ffff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7efe1006ff80 0x7efe10155d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7efe0ffff640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe10070e70 con 0x7efe100c7480 2023-12-04T05:40:07.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7efe0ffff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe10071300 con 0x7efe1014c6c0 2023-12-04T05:40:07.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7efe0ffff640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe100c6330 con 0x7efe1006ff80 2023-12-04T05:40:07.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7efe0effd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efe1014c6c0 0x7efe100c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f3de957b640 1 Processor -- start 2023-12-04T05:40:07.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7efe0effd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efe1014c6c0 0x7efe100c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40788/0 (socket says 172.21.15.107:40788) 2023-12-04T05:40:07.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7efe0effd640 1 -- 172.21.15.107:0/456783264 learned_addr learned my addr 172.21.15.107:0/456783264 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f3de957b640 1 -- start start 2023-12-04T05:40:07.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f87c06fa640 1 Processor -- start 2023-12-04T05:40:07.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f87c06fa640 1 -- start start 2023-12-04T05:40:07.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7efe0effd640 1 -- 172.21.15.107:0/456783264 >> v2:172.21.15.195:3300/0 conn(0x7efe1006ff80 msgr2=0x7efe10155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:07.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7efe0effd640 1 --2- 172.21.15.107:0/456783264 >> v2:172.21.15.195:3300/0 conn(0x7efe1006ff80 0x7efe10155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7efe0effd640 1 -- 172.21.15.107:0/456783264 >> v1:172.21.15.195:6789/0 conn(0x7efe100c7480 legacy=0x7efe1006a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7efe0effd640 1 -- 172.21.15.107:0/456783264 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe100c6610 con 0x7efe1014c6c0 2023-12-04T05:40:07.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ffb6fd54640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffb6814ffb0 0x7ffb6806b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ff018c7f640 1 Processor -- start 2023-12-04T05:40:07.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ffb6fd54640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ffb680c77d0 0x7ffb6806a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f489f59b640 1 Processor -- start 2023-12-04T05:40:07.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ffb6fd54640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb68159ea0 con 0x7ffb68150390 2023-12-04T05:40:07.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ffb6fd54640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb68072420 con 0x7ffb6814ffb0 2023-12-04T05:40:07.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ffb6e2f1640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ffb680c77d0 0x7ffb6806a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ffb6e2f1640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ffb680c77d0 0x7ffb6806a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.107:47112/0 (socket says 172.21.15.107:47112) 2023-12-04T05:40:07.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ffb6e2f1640 1 -- 172.21.15.107:0/840000871 learned_addr learned my addr 172.21.15.107:0/840000871 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ffb6fd54640 1 -- 172.21.15.107:0/840000871 --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb6815a3d0 con 0x7ffb680c77d0 2023-12-04T05:40:07.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f87c06fa640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f87b00dd290 0x7f87b00e66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f87c06fa640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f87b00e6c00 0x7f87b004ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f87c06fa640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b004c660 con 0x7f87b00e71c0 2023-12-04T05:40:07.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f87c06fa640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87b00454b0 con 0x7f87b00e6c00 2023-12-04T05:40:07.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f87c06fa640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b0045790 con 0x7f87b00dd290 2023-12-04T05:40:07.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f489f59b640 1 -- start start 2023-12-04T05:40:07.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7feccf59e640 1 Processor -- start 2023-12-04T05:40:07.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ff018c7f640 1 -- start start 2023-12-04T05:40:07.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7feccf59e640 1 -- start start 2023-12-04T05:40:07.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f3de957b640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3ddc0dd290 0x7f3ddc0e66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f3de957b640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3ddc0e71c0 0x7f3ddc0e6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f3de957b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ddc04c660 con 0x7f3ddc046270 2023-12-04T05:40:07.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f3de957b640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ddc0454b0 con 0x7f3ddc0dd290 2023-12-04T05:40:07.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f3de957b640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc045790 con 0x7f3ddc0e71c0 2023-12-04T05:40:07.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f3de3fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3ddc0dd290 0x7f3ddc0e66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f3de3fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3ddc0dd290 0x7f3ddc0e66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.107:47100/0 (socket says 172.21.15.107:47100) 2023-12-04T05:40:07.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f3de3fff640 1 -- 172.21.15.107:0/96056974 learned_addr learned my addr 172.21.15.107:0/96056974 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ffb6daf0640 1 --2- 172.21.15.107:0/840000871 >> v2:172.21.15.107:3300/0 conn(0x7ffb6814ffb0 0x7ffb6806b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f87bec97640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f87b00e6c00 0x7f87b004ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f87bec97640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f87b00e6c00 0x7f87b004ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.107:47118/0 (socket says 172.21.15.107:47118) 2023-12-04T05:40:07.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f87bec97640 1 -- 172.21.15.107:0/4196566575 learned_addr learned my addr 172.21.15.107:0/4196566575 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f87be496640 1 --2- 172.21.15.107:0/4196566575 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f87b00dd290 0x7f87b00e66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f1122b5c640 1 Processor -- start 2023-12-04T05:40:07.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f1122b5c640 1 -- start start 2023-12-04T05:40:07.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f489f59b640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f489806a5b0 0x7f489806a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f489f59b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f489806b4f0 0x7f489815ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f489f59b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4898162880 con 0x7f48980c5ba0 2023-12-04T05:40:07.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f489f59b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f489806aed0 con 0x7f489806b4f0 2023-12-04T05:40:07.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f489f59b640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f489815f2c0 con 0x7f489806a5b0 2023-12-04T05:40:07.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ff018c7f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff00c0dd290 0x7ff00c0e66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ff018c7f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00c04c660 con 0x7ff00c0e71c0 2023-12-04T05:40:07.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ff018c7f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00c0454b0 con 0x7ff00c0e6c00 2023-12-04T05:40:07.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ff018c7f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff00c045790 con 0x7ff00c0dd290 2023-12-04T05:40:07.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ff013fff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff00c0e6c00 0x7ff00c04ba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:55876/0 (socket says 172.21.15.107:55876) 2023-12-04T05:40:07.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ff013fff640 1 -- 172.21.15.107:0/984004858 learned_addr learned my addr 172.21.15.107:0/984004858 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:07.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ffb6e2f1640 1 -- 172.21.15.107:0/840000871 >> v2:172.21.15.107:3300/0 conn(0x7ffb6814ffb0 msgr2=0x7ffb6806b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:07.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ffb6e2f1640 1 --2- 172.21.15.107:0/840000871 >> v2:172.21.15.107:3300/0 conn(0x7ffb6814ffb0 0x7ffb6806b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ffb6e2f1640 1 -- 172.21.15.107:0/840000871 >> v1:172.21.15.107:6789/0 conn(0x7ffb68150390 legacy=0x7ffb68156840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:40:07.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7ffb6e2f1640 1 -- 172.21.15.107:0/840000871 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb681563b0 con 0x7ffb680c77d0 2023-12-04T05:40:07.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f489d337640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f489806a5b0 0x7f489806a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7feccf59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fecc8004ae0 0x7fecc80338f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7feccf59e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc80dca60 con 0x7fecc80d6cc0 2023-12-04T05:40:07.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7feccf59e640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc80dcd30 con 0x7fecc804c620 2023-12-04T05:40:07.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7feccf59e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecc8007c00 con 0x7fecc8004ae0 2023-12-04T05:40:07.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f489db38640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f48980c5ba0 0x7f48980c6b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:55886/0 (socket says 172.21.15.107:55886) 2023-12-04T05:40:07.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f489db38640 1 -- 172.21.15.107:0/1686426630 learned_addr learned my addr 172.21.15.107:0/1686426630 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:07.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f489cb36640 1 --2- 172.21.15.107:0/1686426630 >> v2:172.21.15.107:3300/0 conn(0x7f489806b4f0 0x7f489815ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7fecced9d640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fecc804c620 0x7fecc80340f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:55892/0 (socket says 172.21.15.107:55892) 2023-12-04T05:40:07.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7fecced9d640 1 -- 172.21.15.107:0/4109883145 learned_addr learned my addr 172.21.15.107:0/4109883145 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:07.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7feccdd9b640 1 --2- 172.21.15.107:0/4109883145 >> v2:172.21.15.107:3300/0 conn(0x7fecc8004ae0 0x7fecc80338f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7ff0137fe640 1 --2- 172.21.15.107:0/984004858 >> v2:172.21.15.107:3300/0 conn(0x7ff00c0dd290 0x7ff00c0e66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f489d337640 1 -- 172.21.15.107:0/1686426630 >> v2:172.21.15.107:3300/0 conn(0x7f489806b4f0 msgr2=0x7f489815ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:07.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f489d337640 1 --2- 172.21.15.107:0/1686426630 >> v2:172.21.15.107:3300/0 conn(0x7f489806b4f0 0x7f489815ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f489d337640 1 -- 172.21.15.107:0/1686426630 >> v1:172.21.15.195:6789/0 conn(0x7f48980c5ba0 legacy=0x7f48980c6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f489d337640 1 -- 172.21.15.107:0/1686426630 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4898158a60 con 0x7f489806a5b0 2023-12-04T05:40:07.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7ff0137fe640 1 -- 172.21.15.107:0/984004858 >> v1:172.21.15.195:6789/0 conn(0x7ff00c0e6c00 legacy=0x7ff00c04ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7ff0137fe640 1 -- 172.21.15.107:0/984004858 >> v1:172.21.15.107:6789/0 conn(0x7ff00c0e71c0 legacy=0x7ff00c046270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:07.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f1122b5c640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f11140d6cc0 0x7f11140338f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7ff0137fe640 1 -- 172.21.15.107:0/984004858 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00c045a70 con 0x7ff00c0dd290 2023-12-04T05:40:07.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f1122b5c640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f111404c620 0x7f11140340f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f1122b5c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11140dca60 con 0x7f1114004ae0 2023-12-04T05:40:07.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f1122b5c640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1114007c00 con 0x7f111404c620 2023-12-04T05:40:07.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f1122b5c640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1114033e30 con 0x7f11140d6cc0 2023-12-04T05:40:07.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7feccd59a640 1 -- 172.21.15.107:0/4109883145 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534783051 0 0) 0x7fecc80dcd30 con 0x7fecc804c620 2023-12-04T05:40:07.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7feccd59a640 1 -- 172.21.15.107:0/4109883145 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecb8003530 con 0x7fecc804c620 2023-12-04T05:40:07.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7feccd59a640 1 -- 172.21.15.107:0/4109883145 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275876203 0 0) 0x7fecc80dca60 con 0x7fecc80d6cc0 2023-12-04T05:40:07.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7feccd59a640 1 -- 172.21.15.107:0/4109883145 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc80dcd30 con 0x7fecc80d6cc0 2023-12-04T05:40:07.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f11208f8640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f11140d6cc0 0x7f11140338f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f11208f8640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f11140d6cc0 0x7f11140338f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.107:42532/0 (socket says 172.21.15.107:42532) 2023-12-04T05:40:07.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f11210f9640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f111404c620 0x7f11140340f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f11208f8640 1 -- 172.21.15.107:0/2286247660 learned_addr learned my addr 172.21.15.107:0/2286247660 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f3de3fff640 1 -- 172.21.15.107:0/96056974 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3ddc0e71c0 msgr2=0x7f3ddc0e6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:07.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f3de3fff640 1 --2- 172.21.15.107:0/96056974 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3ddc0e71c0 0x7f3ddc0e6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f3de3fff640 1 -- 172.21.15.107:0/96056974 >> v1:172.21.15.107:6789/0 conn(0x7f3ddc046270 legacy=0x7f3ddc04ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:07.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f3de3fff640 1 -- 172.21.15.107:0/96056974 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ddc045a70 con 0x7f3ddc0dd290 2023-12-04T05:40:07.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f11208f8640 1 -- 172.21.15.107:0/2286247660 >> v2:172.21.15.195:3300/0 conn(0x7f111404c620 msgr2=0x7f11140340f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:07.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f11208f8640 1 --2- 172.21.15.107:0/2286247660 >> v2:172.21.15.195:3300/0 conn(0x7f111404c620 0x7f11140340f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f11208f8640 1 -- 172.21.15.107:0/2286247660 >> v1:172.21.15.107:6789/0 conn(0x7f1114004ae0 legacy=0x7f111404bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:07.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f11208f8640 1 -- 172.21.15.107:0/2286247660 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11140dcd30 con 0x7f11140d6cc0 2023-12-04T05:40:07.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7feccd59a640 1 -- 172.21.15.107:0/4109883145 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1713570231 0 0) 0x7fecb8003530 con 0x7fecc804c620 2023-12-04T05:40:07.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7feccd59a640 1 -- 172.21.15.107:0/4109883145 >> v2:172.21.15.107:3300/0 conn(0x7fecc8004ae0 msgr2=0x7fecc80338f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:07.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7feccd59a640 1 --2- 172.21.15.107:0/4109883145 >> v2:172.21.15.107:3300/0 conn(0x7fecc8004ae0 0x7fecc80338f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7feccd59a640 1 -- 172.21.15.107:0/4109883145 >> v1:172.21.15.107:6789/0 conn(0x7fecc80d6cc0 legacy=0x7fecc804bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7feccd59a640 1 -- 172.21.15.107:0/4109883145 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc80dd010 con 0x7fecc804c620 2023-12-04T05:40:07.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7feccd59a640 1 -- 172.21.15.107:0/4109883145 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7fecd004edb0 con 0x7fecc804c620 2023-12-04T05:40:07.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7feccd59a640 1 -- 172.21.15.107:0/4109883145 >> v1:172.21.15.195:6789/0 conn(0x7fecc804c620 legacy=0x7fecc80340f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7feccd59a640 1 --2- 172.21.15.107:0/4109883145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecb8003b10 0x7fecb8003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7feccd59a640 1 --2- 172.21.15.107:0/4109883145 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fecb80047d0 0x7fecb8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7feccd59a640 1 --2- 172.21.15.107:0/4109883145 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fecb8005190 0x7fecb800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7feccd59a640 1 -- 172.21.15.107:0/4109883145 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc8007c00 con 0x7fecb8003b10 2023-12-04T05:40:07.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7feccd59a640 1 -- 172.21.15.107:0/4109883145 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb800bed0 con 0x7fecb80047d0 2023-12-04T05:40:07.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7feccd59a640 1 -- 172.21.15.107:0/4109883145 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecb800c010 con 0x7fecb8005190 2023-12-04T05:40:07.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f87bec97640 1 -- 172.21.15.107:0/4196566575 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f87b00dd290 msgr2=0x7f87b00e66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:07.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f87bec97640 1 --2- 172.21.15.107:0/4196566575 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f87b00dd290 0x7f87b00e66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f87bec97640 1 -- 172.21.15.107:0/4196566575 >> v1:172.21.15.107:6789/0 conn(0x7f87b00e71c0 legacy=0x7f87b0046270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:40:07.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.691+0000 7f87bec97640 1 -- 172.21.15.107:0/4196566575 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b0045a70 con 0x7f87b00e6c00 2023-12-04T05:40:07.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7ffb6e2f1640 1 --2- 172.21.15.107:0/840000871 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb680c77d0 0x7ffb6806a5b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffb68073190 tx=0x7ffb6001cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e27b25a8fbe2184d server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7feccdd9b640 1 --2- 172.21.15.107:0/4109883145 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fecb80047d0 0x7fecb8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7fecced9d640 1 --2- 172.21.15.107:0/4109883145 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fecb8005190 0x7fecb800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7efe0effd640 1 --2- 172.21.15.107:0/456783264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe1014c6c0 0x7efe100c5ca0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efe1006e0a0 tx=0x7efe0001d700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad93958c32329db4 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f87bec97640 1 --2- 172.21.15.107:0/4196566575 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87b00e6c00 0x7f87b004ba40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f87b807b5e0 tx=0x7f87b809a190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2003dec33c501b4 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7fecce59c640 1 --2- 172.21.15.107:0/4109883145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecb8003b10 0x7fecb8003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7efe0dffb640 1 -- 172.21.15.107:0/456783264 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe000a67b0 con 0x7efe1014c6c0 2023-12-04T05:40:07.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f3de3fff640 1 --2- 172.21.15.107:0/96056974 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3ddc0dd290 0x7f3ddc0e66c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3de406a260 tx=0x7f3de4099e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5aa835db4830478a server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7fecce59c640 1 -- 172.21.15.107:0/4109883145 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fecb8005190 msgr2=0x7fecb800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:07.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7fecce59c640 1 --2- 172.21.15.107:0/4109883145 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fecb8005190 0x7fecb800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7fecce59c640 1 -- 172.21.15.107:0/4109883145 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fecb80047d0 msgr2=0x7fecb8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:07.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7fecce59c640 1 --2- 172.21.15.107:0/4109883145 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fecb80047d0 0x7fecb8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7fecce59c640 1 -- 172.21.15.107:0/4109883145 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc80dd010 con 0x7fecb8003b10 2023-12-04T05:40:07.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7fecced9d640 1 -- 172.21.15.107:0/4109883145 reap_dead start 2023-12-04T05:40:07.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f87bd494640 1 -- 172.21.15.107:0/4196566575 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b8123e50 con 0x7f87b00e6c00 2023-12-04T05:40:07.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f87bd494640 1 -- 172.21.15.107:0/4196566575 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b812b440 con 0x7f87b00e6c00 2023-12-04T05:40:07.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f87bd494640 1 -- 172.21.15.107:0/4196566575 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b812b870 con 0x7f87b00e6c00 2023-12-04T05:40:07.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7efe0dffb640 1 -- 172.21.15.107:0/456783264 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe000a6910 con 0x7efe1014c6c0 2023-12-04T05:40:07.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7efe0dffb640 1 -- 172.21.15.107:0/456783264 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe000a6ca0 con 0x7efe1014c6c0 2023-12-04T05:40:07.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7efe0ffff640 1 -- 172.21.15.107:0/456783264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe1014c6c0 msgr2=0x7efe100c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7efe0ffff640 1 --2- 172.21.15.107:0/456783264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe1014c6c0 0x7efe100c5ca0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efe1006e0a0 tx=0x7efe0001d700 comp rx=0 tx=0).stop 2023-12-04T05:40:07.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f3de2ffd640 1 -- 172.21.15.107:0/96056974 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de4123af0 con 0x7f3ddc0dd290 2023-12-04T05:40:07.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f3de2ffd640 1 -- 172.21.15.107:0/96056974 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de4123c50 con 0x7f3ddc0dd290 2023-12-04T05:40:07.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f3de2ffd640 1 -- 172.21.15.107:0/96056974 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de412b640 con 0x7f3ddc0dd290 2023-12-04T05:40:07.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f489d337640 1 --2- 172.21.15.107:0/1686426630 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f489806a5b0 0x7f489806a990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f489806e410 tx=0x7f488801ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36586b77f48e33bb server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f488ffff640 1 -- 172.21.15.107:0/1686426630 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48880a8950 con 0x7f489806a5b0 2023-12-04T05:40:07.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f488ffff640 1 -- 172.21.15.107:0/1686426630 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48880a8ab0 con 0x7f489806a5b0 2023-12-04T05:40:07.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.695+0000 7f488ffff640 1 -- 172.21.15.107:0/1686426630 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48880a8e40 con 0x7f489806a5b0 2023-12-04T05:40:07.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f87c06fa640 1 -- 172.21.15.107:0/4196566575 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87b00e6c00 msgr2=0x7f87b004ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f87c06fa640 1 --2- 172.21.15.107:0/4196566575 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87b00e6c00 0x7f87b004ba40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f87b807b5e0 tx=0x7f87b809a190 comp rx=0 tx=0).stop 2023-12-04T05:40:07.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7efe0ffff640 1 -- 172.21.15.107:0/456783264 shutdown_connections 2023-12-04T05:40:07.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7efe0ffff640 1 --2- 172.21.15.107:0/456783264 >> v2:172.21.15.195:3300/0 conn(0x7efe1006ff80 0x7efe10155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7efe0ffff640 1 --2- 172.21.15.107:0/456783264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe1014c6c0 0x7efe100c5ca0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7efe0ffff640 1 -- 172.21.15.107:0/456783264 >> 172.21.15.107:0/456783264 conn(0x7efe100cb3a0 msgr2=0x7efe100cb7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7efe0ffff640 1 -- 172.21.15.107:0/456783264 shutdown_connections 2023-12-04T05:40:07.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7ffb6caee640 1 -- 172.21.15.107:0/840000871 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb600a5c00 con 0x7ffb680c77d0 2023-12-04T05:40:07.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7ffb6caee640 1 -- 172.21.15.107:0/840000871 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb600a5d60 con 0x7ffb680c77d0 2023-12-04T05:40:07.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7ff0137fe640 1 --2- 172.21.15.107:0/984004858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff00c0dd290 0x7ff00c0e66c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff008009740 tx=0x7ff00801c2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dd2bbcd95052434 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7ffb6caee640 1 -- 172.21.15.107:0/840000871 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6000a2f0 con 0x7ffb680c77d0 2023-12-04T05:40:07.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f11208f8640 1 --2- 172.21.15.107:0/2286247660 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f11140d6cc0 0x7f11140338f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1110009870 tx=0x7f111001ec30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8729622bbf34f942 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f489f59b640 1 -- 172.21.15.107:0/1686426630 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f489806a5b0 msgr2=0x7f489806a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f489f59b640 1 --2- 172.21.15.107:0/1686426630 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f489806a5b0 0x7f489806a990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f489806e410 tx=0x7f488801ec50 comp rx=0 tx=0).stop 2023-12-04T05:40:07.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f3de957b640 1 -- 172.21.15.107:0/96056974 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3ddc0dd290 msgr2=0x7f3ddc0e66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f3de957b640 1 --2- 172.21.15.107:0/96056974 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3ddc0dd290 0x7f3ddc0e66c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3de406a260 tx=0x7f3de4099e10 comp rx=0 tx=0).stop 2023-12-04T05:40:07.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f87c06fa640 1 -- 172.21.15.107:0/4196566575 shutdown_connections 2023-12-04T05:40:07.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f87c06fa640 1 --2- 172.21.15.107:0/4196566575 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87b00e6c00 0x7f87b004ba40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f87c06fa640 1 --2- 172.21.15.107:0/4196566575 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f87b00dd290 0x7f87b00e66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f87c06fa640 1 -- 172.21.15.107:0/4196566575 >> 172.21.15.107:0/4196566575 conn(0x7f87b0050160 msgr2=0x7f87b0050570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f3de957b640 1 -- 172.21.15.107:0/96056974 shutdown_connections 2023-12-04T05:40:07.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f3de957b640 1 --2- 172.21.15.107:0/96056974 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3ddc0e71c0 0x7f3ddc0e6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f3de957b640 1 --2- 172.21.15.107:0/96056974 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3ddc0dd290 0x7f3ddc0e66c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f3de957b640 1 -- 172.21.15.107:0/96056974 >> 172.21.15.107:0/96056974 conn(0x7f3ddc050160 msgr2=0x7f3ddc050570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f489f59b640 1 -- 172.21.15.107:0/1686426630 shutdown_connections 2023-12-04T05:40:07.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f489f59b640 1 --2- 172.21.15.107:0/1686426630 >> v2:172.21.15.107:3300/0 conn(0x7f489806b4f0 0x7f489815ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f489f59b640 1 --2- 172.21.15.107:0/1686426630 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f489806a5b0 0x7f489806a990 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f489f59b640 1 -- 172.21.15.107:0/1686426630 >> 172.21.15.107:0/1686426630 conn(0x7f48980cb2a0 msgr2=0x7f48980cb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f111b7fe640 1 -- 172.21.15.107:0/2286247660 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11100a88f0 con 0x7f11140d6cc0 2023-12-04T05:40:07.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f111b7fe640 1 -- 172.21.15.107:0/2286247660 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11100a8a50 con 0x7f11140d6cc0 2023-12-04T05:40:07.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f111b7fe640 1 -- 172.21.15.107:0/2286247660 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11100a8de0 con 0x7f11140d6cc0 2023-12-04T05:40:07.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7efe0ffff640 1 -- 172.21.15.107:0/456783264 wait complete. 2023-12-04T05:40:07.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7efe0ffff640 1 Processor -- start 2023-12-04T05:40:07.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7ffb6fd54640 1 -- 172.21.15.107:0/840000871 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb680c77d0 msgr2=0x7ffb6806a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7ffb6fd54640 1 --2- 172.21.15.107:0/840000871 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb680c77d0 0x7ffb6806a5b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffb68073190 tx=0x7ffb6001cd50 comp rx=0 tx=0).stop 2023-12-04T05:40:07.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7ffb6fd54640 1 -- 172.21.15.107:0/840000871 shutdown_connections 2023-12-04T05:40:07.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7ffb6fd54640 1 --2- 172.21.15.107:0/840000871 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb680c77d0 0x7ffb6806a5b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7ffb6fd54640 1 --2- 172.21.15.107:0/840000871 >> v2:172.21.15.107:3300/0 conn(0x7ffb6814ffb0 0x7ffb6806b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7ffb6fd54640 1 -- 172.21.15.107:0/840000871 >> 172.21.15.107:0/840000871 conn(0x7ffb680cb2a0 msgr2=0x7ffb680cb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7ffb6fd54640 1 -- 172.21.15.107:0/840000871 shutdown_connections 2023-12-04T05:40:07.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7ffb6fd54640 1 -- 172.21.15.107:0/840000871 wait complete. 2023-12-04T05:40:07.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7fecce59c640 1 --2- 172.21.15.107:0/4109883145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecb8003b10 0x7fecb8003f10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fecc00018b0 tx=0x7fecc001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c964fb5ae3e922e server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f87c06fa640 1 -- 172.21.15.107:0/4196566575 shutdown_connections 2023-12-04T05:40:07.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f3de957b640 1 -- 172.21.15.107:0/96056974 shutdown_connections 2023-12-04T05:40:07.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f3de957b640 1 -- 172.21.15.107:0/96056974 wait complete. 2023-12-04T05:40:07.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f489f59b640 1 -- 172.21.15.107:0/1686426630 shutdown_connections 2023-12-04T05:40:07.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f1122b5c640 1 -- 172.21.15.107:0/2286247660 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f11140d6cc0 msgr2=0x7f11140338f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f1122b5c640 1 --2- 172.21.15.107:0/2286247660 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f11140d6cc0 0x7f11140338f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1110009870 tx=0x7f111001ec30 comp rx=0 tx=0).stop 2023-12-04T05:40:07.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7ff0127fc640 1 -- 172.21.15.107:0/984004858 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0080ab600 con 0x7ff00c0dd290 2023-12-04T05:40:07.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f1122b5c640 1 -- 172.21.15.107:0/2286247660 shutdown_connections 2023-12-04T05:40:07.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f1122b5c640 1 --2- 172.21.15.107:0/2286247660 >> v2:172.21.15.195:3300/0 conn(0x7f111404c620 0x7f11140340f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f1122b5c640 1 --2- 172.21.15.107:0/2286247660 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f11140d6cc0 0x7f11140338f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f1122b5c640 1 -- 172.21.15.107:0/2286247660 >> 172.21.15.107:0/2286247660 conn(0x7f11140505b0 msgr2=0x7f11140509c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f87c06fa640 1 -- 172.21.15.107:0/4196566575 wait complete. 2023-12-04T05:40:07.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f1122b5c640 1 -- 172.21.15.107:0/2286247660 shutdown_connections 2023-12-04T05:40:07.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7efe0ffff640 1 -- start start 2023-12-04T05:40:07.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7feccd59a640 1 -- 172.21.15.107:0/4109883145 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc0005270 con 0x7fecb8003b10 2023-12-04T05:40:07.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7feccd59a640 1 -- 172.21.15.107:0/4109883145 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc00053d0 con 0x7fecb8003b10 2023-12-04T05:40:07.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7feccd59a640 1 -- 172.21.15.107:0/4109883145 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc00a88f0 con 0x7fecb8003b10 2023-12-04T05:40:07.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f3de957b640 1 Processor -- start 2023-12-04T05:40:07.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f489f59b640 1 -- 172.21.15.107:0/1686426630 wait complete. 2023-12-04T05:40:07.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7ff0127fc640 1 -- 172.21.15.107:0/984004858 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0080a7050 con 0x7ff00c0dd290 2023-12-04T05:40:07.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f1122b5c640 1 -- 172.21.15.107:0/2286247660 wait complete. 2023-12-04T05:40:07.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7f1122b5c640 1 Processor -- start 2023-12-04T05:40:07.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7feccf59e640 1 -- 172.21.15.107:0/4109883145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecb8003b10 msgr2=0x7fecb8003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7feccf59e640 1 --2- 172.21.15.107:0/4109883145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecb8003b10 0x7fecb8003f10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fecc00018b0 tx=0x7fecc001ec90 comp rx=0 tx=0).stop 2023-12-04T05:40:07.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7feccf59e640 1 -- 172.21.15.107:0/4109883145 shutdown_connections 2023-12-04T05:40:07.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7feccf59e640 1 --2- 172.21.15.107:0/4109883145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecb8003b10 0x7fecb8003f10 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7feccf59e640 1 -- 172.21.15.107:0/4109883145 >> 172.21.15.107:0/4109883145 conn(0x7fecc80505b0 msgr2=0x7fecc80509c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7feccf59e640 1 -- 172.21.15.107:0/4109883145 shutdown_connections 2023-12-04T05:40:07.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7efe0ffff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efe1006ff80 0x7efe101ff780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7efe0ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe100c7480 0x7efe101598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7efe0ffff640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efe1014c6c0 0x7efe10149f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7efe0ffff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe10157e10 con 0x7efe100c7480 2023-12-04T05:40:07.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7efe0ffff640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe10156f60 con 0x7efe1014c6c0 2023-12-04T05:40:07.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7efe0ffff640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe10157200 con 0x7efe1006ff80 2023-12-04T05:40:07.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f1122b5c640 1 -- start start 2023-12-04T05:40:07.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7efe0e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe100c7480 0x7efe101598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7efe0e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe100c7480 0x7efe101598c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40826/0 (socket says 172.21.15.107:40826) 2023-12-04T05:40:07.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7efe0e7fc640 1 -- 172.21.15.107:0/3843017370 learned_addr learned my addr 172.21.15.107:0/3843017370 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7efe0effd640 1 --2- 172.21.15.107:0/3843017370 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efe1006ff80 0x7efe101ff780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7efe0f7fe640 1 --2- 172.21.15.107:0/3843017370 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efe1014c6c0 0x7efe10149f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f489f59b640 1 Processor -- start 2023-12-04T05:40:07.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ff018c7f640 1 -- 172.21.15.107:0/984004858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff00c0dd290 msgr2=0x7ff00c0e66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ff018c7f640 1 --2- 172.21.15.107:0/984004858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff00c0dd290 0x7ff00c0e66c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff008009740 tx=0x7ff00801c2e0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ffb6fd54640 1 Processor -- start 2023-12-04T05:40:07.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.699+0000 7feccf59e640 1 -- 172.21.15.107:0/4109883145 wait complete. 2023-12-04T05:40:07.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ffb6fd54640 1 -- start start 2023-12-04T05:40:07.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f87c06fa640 1 Processor -- start 2023-12-04T05:40:07.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7efe0e7fc640 1 -- 172.21.15.107:0/3843017370 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efe1006ff80 msgr2=0x7efe101ff780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7efe0e7fc640 1 --2- 172.21.15.107:0/3843017370 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efe1006ff80 0x7efe101ff780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7efe0e7fc640 1 -- 172.21.15.107:0/3843017370 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efe1014c6c0 msgr2=0x7efe10149f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7efe0e7fc640 1 --2- 172.21.15.107:0/3843017370 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efe1014c6c0 0x7efe10149f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7efe0e7fc640 1 -- 172.21.15.107:0/3843017370 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe00009d60 con 0x7efe100c7480 2023-12-04T05:40:07.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7efe0e7fc640 1 --2- 172.21.15.107:0/3843017370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe100c7480 0x7efe101598c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efe0400b480 tx=0x7efe0400b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f489f59b640 1 -- start start 2023-12-04T05:40:07.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f489f59b640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f489806a5b0 0x7f489806d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f489f59b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f489806b4f0 0x7f48980bce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f489f59b640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48980c5ba0 0x7f48980bd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f489f59b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48980bc8c0 con 0x7f489806b4f0 2023-12-04T05:40:07.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f489f59b640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48980bbd40 con 0x7f48980c5ba0 2023-12-04T05:40:07.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f489f59b640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48980bc020 con 0x7f489806a5b0 2023-12-04T05:40:07.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f489db38640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48980c5ba0 0x7f48980bd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f489cb36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f489806b4f0 0x7f48980bce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f489db38640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48980c5ba0 0x7f48980bd3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.107:47160/0 (socket says 172.21.15.107:47160) 2023-12-04T05:40:07.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f489db38640 1 -- 172.21.15.107:0/685017628 learned_addr learned my addr 172.21.15.107:0/685017628 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f489d337640 1 --2- 172.21.15.107:0/685017628 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f489806a5b0 0x7f489806d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ff018c7f640 1 -- 172.21.15.107:0/984004858 shutdown_connections 2023-12-04T05:40:07.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ff018c7f640 1 --2- 172.21.15.107:0/984004858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff00c0dd290 0x7ff00c0e66c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ff018c7f640 1 -- 172.21.15.107:0/984004858 >> 172.21.15.107:0/984004858 conn(0x7ff00c050160 msgr2=0x7ff00c050570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:07.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ff018c7f640 1 -- 172.21.15.107:0/984004858 shutdown_connections 2023-12-04T05:40:07.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f1122b5c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1114004ae0 0x7f11140d9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f1122b5c640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f111404c620 0x7f11140db970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f1122b5c640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f11140d6cc0 0x7f11140cd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f1122b5c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11140cd440 con 0x7f1114004ae0 2023-12-04T05:40:07.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f1122b5c640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11140cce10 con 0x7f111404c620 2023-12-04T05:40:07.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f1122b5c640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11140d9b00 con 0x7f11140d6cc0 2023-12-04T05:40:07.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f11208f8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1114004ae0 0x7f11140d9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f11208f8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1114004ae0 0x7f11140d9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40842/0 (socket says 172.21.15.107:40842) 2023-12-04T05:40:07.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f11208f8640 1 -- 172.21.15.107:0/468814680 learned_addr learned my addr 172.21.15.107:0/468814680 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f11210f9640 1 --2- 172.21.15.107:0/468814680 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f11140d6cc0 0x7f11140cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f111bfff640 1 --2- 172.21.15.107:0/468814680 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f111404c620 0x7f11140db970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f11208f8640 1 -- 172.21.15.107:0/468814680 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f11140d6cc0 msgr2=0x7f11140cd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f11208f8640 1 --2- 172.21.15.107:0/468814680 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f11140d6cc0 0x7f11140cd860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f11208f8640 1 -- 172.21.15.107:0/468814680 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f111404c620 msgr2=0x7f11140db970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f11208f8640 1 --2- 172.21.15.107:0/468814680 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f111404c620 0x7f11140db970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f11208f8640 1 -- 172.21.15.107:0/468814680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11100094d0 con 0x7f1114004ae0 2023-12-04T05:40:07.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7feccf59e640 1 Processor -- start 2023-12-04T05:40:07.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f87c06fa640 1 -- start start 2023-12-04T05:40:07.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f87c06fa640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87b00dd290 0x7f87b00499e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f87c06fa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87b00e6c00 0x7f87b00dad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f87c06fa640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f87b00e71c0 0x7f87b01789f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f87c06fa640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b0049fe0 con 0x7f87b00e6c00 2023-12-04T05:40:07.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f87c06fa640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b017c000 con 0x7f87b00dd290 2023-12-04T05:40:07.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f87c06fa640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b017c2e0 con 0x7f87b00e71c0 2023-12-04T05:40:07.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f87be496640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87b00dd290 0x7f87b00499e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f87be496640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87b00dd290 0x7f87b00499e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.107:47176/0 (socket says 172.21.15.107:47176) 2023-12-04T05:40:07.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f87be496640 1 -- 172.21.15.107:0/2085076697 learned_addr learned my addr 172.21.15.107:0/2085076697 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f489cb36640 1 -- 172.21.15.107:0/685017628 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f489806a5b0 msgr2=0x7f489806d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f489cb36640 1 --2- 172.21.15.107:0/685017628 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f489806a5b0 0x7f489806d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f489cb36640 1 -- 172.21.15.107:0/685017628 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48980c5ba0 msgr2=0x7f48980bd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f489cb36640 1 --2- 172.21.15.107:0/685017628 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48980c5ba0 0x7f48980bd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f489cb36640 1 -- 172.21.15.107:0/685017628 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4888009420 con 0x7f489806b4f0 2023-12-04T05:40:07.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f87bec97640 1 --2- 172.21.15.107:0/2085076697 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f87b00e71c0 0x7f87b01789f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f11208f8640 1 --2- 172.21.15.107:0/468814680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1114004ae0 0x7f11140d9e00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f11100b3040 tx=0x7f111001b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7feccf59e640 1 -- start start 2023-12-04T05:40:07.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7feccf59e640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fecb8003b10 0x7fecc8172130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ff018c7f640 1 -- 172.21.15.107:0/984004858 wait complete. 2023-12-04T05:40:07.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ffb6fd54640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb680c77d0 0x7ffb6814b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7feccf59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecc8164e10 0x7fecc8005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7feccf59e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fecc81633a0 0x7fecc8163820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7feccf59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc8163d60 con 0x7fecc8164e10 2023-12-04T05:40:07.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7feccf59e640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc80d7740 con 0x7fecb8003b10 2023-12-04T05:40:07.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7feccf59e640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc80d7a00 con 0x7fecc81633a0 2023-12-04T05:40:07.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ff018c7f640 1 Processor -- start 2023-12-04T05:40:07.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ffb6fd54640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb6814ffb0 0x7ffb681f2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ffb6fd54640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb68150390 0x7ffb680c9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f87be496640 1 -- 172.21.15.107:0/2085076697 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f87b00e71c0 msgr2=0x7f87b01789f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f87be496640 1 --2- 172.21.15.107:0/2085076697 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f87b00e71c0 0x7f87b01789f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f87be496640 1 -- 172.21.15.107:0/2085076697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87b00e6c00 msgr2=0x7f87b00dad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f87be496640 1 --2- 172.21.15.107:0/2085076697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87b00e6c00 0x7f87b00dad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f87be496640 1 -- 172.21.15.107:0/2085076697 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b017c5c0 con 0x7f87b00dd290 2023-12-04T05:40:07.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7fecced9d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fecc81633a0 0x7fecc8163820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f3de957b640 1 -- start start 2023-12-04T05:40:07.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f3de957b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ddc046270 0x7f3ddc16fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f3de957b640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3ddc0dd290 0x7f3ddc17c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f3de957b640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3ddc0e71c0 0x7f3ddc0d5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f3de957b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc17cf10 con 0x7f3ddc046270 2023-12-04T05:40:07.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f3de957b640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc0d8c10 con 0x7f3ddc0dd290 2023-12-04T05:40:07.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f3de957b640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc178a50 con 0x7f3ddc0e71c0 2023-12-04T05:40:07.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7fecced9d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fecc81633a0 0x7fecc8163820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.107:42590/0 (socket says 172.21.15.107:42590) 2023-12-04T05:40:07.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7fecced9d640 1 -- 172.21.15.107:0/1920704216 learned_addr learned my addr 172.21.15.107:0/1920704216 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f489cb36640 1 --2- 172.21.15.107:0/685017628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f489806b4f0 0x7f48980bce70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f489400e870 tx=0x7f489400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f11197fa640 1 -- 172.21.15.107:0/468814680 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11100b9930 con 0x7f1114004ae0 2023-12-04T05:40:07.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f1122b5c640 1 -- 172.21.15.107:0/468814680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11140ca590 con 0x7f1114004ae0 2023-12-04T05:40:07.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f11197fa640 1 -- 172.21.15.107:0/468814680 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11100b9ad0 con 0x7f1114004ae0 2023-12-04T05:40:07.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f11197fa640 1 -- 172.21.15.107:0/468814680 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11100b9e80 con 0x7f1114004ae0 2023-12-04T05:40:07.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f1122b5c640 1 -- 172.21.15.107:0/468814680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11140cd060 con 0x7f1114004ae0 2023-12-04T05:40:07.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ffb6fd54640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb681f1ed0 con 0x7ffb6814ffb0 2023-12-04T05:40:07.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7fecce59c640 1 --2- 172.21.15.107:0/1920704216 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fecb8003b10 0x7fecc8172130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ffb6daf0640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb680c77d0 0x7ffb6814b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ffb6daf0640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb680c77d0 0x7ffb6814b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.107:42592/0 (socket says 172.21.15.107:42592) 2023-12-04T05:40:07.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ffb6daf0640 1 -- 172.21.15.107:0/3241084504 learned_addr learned my addr 172.21.15.107:0/3241084504 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f87be496640 1 --2- 172.21.15.107:0/2085076697 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87b00dd290 0x7f87b00499e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f87a800c7f0 tx=0x7f87a800ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f3de8d7a640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3ddc0e71c0 0x7f3ddc0d5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f3de8d7a640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3ddc0e71c0 0x7f3ddc0d5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.107:42604/0 (socket says 172.21.15.107:42604) 2023-12-04T05:40:07.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f3de8d7a640 1 -- 172.21.15.107:0/4268987116 learned_addr learned my addr 172.21.15.107:0/4268987116 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f488dffb640 1 -- 172.21.15.107:0/685017628 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48940095d0 con 0x7f489806b4f0 2023-12-04T05:40:07.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f488dffb640 1 -- 172.21.15.107:0/685017628 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f489400c940 con 0x7f489806b4f0 2023-12-04T05:40:07.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f488dffb640 1 -- 172.21.15.107:0/685017628 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489400ccb0 con 0x7f489806b4f0 2023-12-04T05:40:07.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ffb6fd54640 1 -- 172.21.15.107:0/3241084504 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb681f2490 con 0x7ffb68150390 2023-12-04T05:40:07.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ffb6e2f1640 1 --2- 172.21.15.107:0/3241084504 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb68150390 0x7ffb680c9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ff018c7f640 1 -- start start 2023-12-04T05:40:07.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ff018c7f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff00c0dd290 0x7ff00c1744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ff018c7f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff00c0e6c00 0x7ff00c174a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ff018c7f640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff00c0e71c0 0x7ff00c0d5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ff018c7f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c182190 con 0x7ff00c0dd290 2023-12-04T05:40:07.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ff018c7f640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c182680 con 0x7ff00c0e71c0 2023-12-04T05:40:07.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7feccdd9b640 1 --2- 172.21.15.107:0/1920704216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecc8164e10 0x7fecc8005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ff018c7f640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c0d9fc0 con 0x7ff00c0e6c00 2023-12-04T05:40:07.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7ffb6fd54640 1 -- 172.21.15.107:0/3241084504 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6814af60 con 0x7ffb680c77d0 2023-12-04T05:40:07.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f489f59b640 1 -- 172.21.15.107:0/685017628 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48980bc300 con 0x7f489806b4f0 2023-12-04T05:40:07.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7f489f59b640 1 -- 172.21.15.107:0/685017628 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48980bea20 con 0x7f489806b4f0 2023-12-04T05:40:07.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7ff012ffd640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff00c0e6c00 0x7ff00c174a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7ff012ffd640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff00c0e6c00 0x7ff00c174a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.107:42618/0 (socket says 172.21.15.107:42618) 2023-12-04T05:40:07.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7ff012ffd640 1 -- 172.21.15.107:0/285567384 learned_addr learned my addr 172.21.15.107:0/285567384 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:07.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7ff013fff640 1 --2- 172.21.15.107:0/285567384 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff00c0e71c0 0x7ff00c0d5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7ff0137fe640 1 --2- 172.21.15.107:0/285567384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff00c0dd290 0x7ff00c1744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7f1122b5c640 1 -- 172.21.15.107:0/468814680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1114005f80 con 0x7f1114004ae0 2023-12-04T05:40:07.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7ffb6d2ef640 1 --2- 172.21.15.107:0/3241084504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb6814ffb0 0x7ffb681f2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7ffb6d2ef640 1 -- 172.21.15.107:0/3241084504 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb680c77d0 msgr2=0x7ffb6814b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7ffb6d2ef640 1 --2- 172.21.15.107:0/3241084504 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb680c77d0 0x7ffb6814b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7ffb6d2ef640 1 -- 172.21.15.107:0/3241084504 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb68150390 msgr2=0x7ffb680c9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7ffb6d2ef640 1 --2- 172.21.15.107:0/3241084504 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb68150390 0x7ffb680c9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7ffb6d2ef640 1 -- 172.21.15.107:0/3241084504 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6000c050 con 0x7ffb6814ffb0 2023-12-04T05:40:07.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7ffb6d2ef640 1 --2- 172.21.15.107:0/3241084504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb6814ffb0 0x7ffb681f2920 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffb5800b3e0 tx=0x7ffb5800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7fecced9d640 1 -- 172.21.15.107:0/1920704216 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fecb8003b10 msgr2=0x7fecc8172130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7fecced9d640 1 --2- 172.21.15.107:0/1920704216 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fecb8003b10 0x7fecc8172130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7fecced9d640 1 -- 172.21.15.107:0/1920704216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecc8164e10 msgr2=0x7fecc8005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7fecced9d640 1 --2- 172.21.15.107:0/1920704216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecc8164e10 0x7fecc8005ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7fecced9d640 1 -- 172.21.15.107:0/1920704216 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc80d7ce0 con 0x7fecc81633a0 2023-12-04T05:40:07.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7fecced9d640 1 --2- 172.21.15.107:0/1920704216 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fecc81633a0 0x7fecc8163820 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fecd004fba0 tx=0x7fecd0062130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7f87aeffd640 1 -- 172.21.15.107:0/2085076697 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8007820 con 0x7f87b00dd290 2023-12-04T05:40:07.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7f87aeffd640 1 -- 172.21.15.107:0/2085076697 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a8007980 con 0x7f87b00dd290 2023-12-04T05:40:07.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7f87aeffd640 1 -- 172.21.15.107:0/2085076697 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8007d30 con 0x7f87b00dd290 2023-12-04T05:40:07.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7f87c06fa640 1 -- 172.21.15.107:0/2085076697 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b017c8a0 con 0x7f87b00dd290 2023-12-04T05:40:07.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7f87c06fa640 1 -- 172.21.15.107:0/2085076697 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b004c2d0 con 0x7f87b00dd290 2023-12-04T05:40:07.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.703+0000 7f3de37fe640 1 --2- 172.21.15.107:0/4268987116 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3ddc0dd290 0x7f3ddc17c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7f3de3fff640 1 --2- 172.21.15.107:0/4268987116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ddc046270 0x7f3ddc16fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7f3de8d7a640 1 -- 172.21.15.107:0/4268987116 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3ddc0dd290 msgr2=0x7f3ddc17c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7f3de8d7a640 1 --2- 172.21.15.107:0/4268987116 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3ddc0dd290 0x7f3ddc17c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7f3de8d7a640 1 -- 172.21.15.107:0/4268987116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ddc046270 msgr2=0x7f3ddc16fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7f3de8d7a640 1 --2- 172.21.15.107:0/4268987116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ddc046270 0x7f3ddc16fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7f3de8d7a640 1 -- 172.21.15.107:0/4268987116 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ddc0d8eb0 con 0x7f3ddc0e71c0 2023-12-04T05:40:07.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7f3de8d7a640 1 --2- 172.21.15.107:0/4268987116 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3ddc0e71c0 0x7f3ddc0d5fc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3dd000b410 tx=0x7f3dd000b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7ff012ffd640 1 -- 172.21.15.107:0/285567384 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff00c0e71c0 msgr2=0x7ff00c0d5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7ff012ffd640 1 --2- 172.21.15.107:0/285567384 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff00c0e71c0 0x7ff00c0d5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7ff012ffd640 1 -- 172.21.15.107:0/285567384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff00c0dd290 msgr2=0x7ff00c1744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:07.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7ff012ffd640 1 --2- 172.21.15.107:0/285567384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff00c0dd290 0x7ff00c1744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:07.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7ff012ffd640 1 -- 172.21.15.107:0/285567384 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff008009420 con 0x7ff00c0e6c00 2023-12-04T05:40:07.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7fecbeffd640 1 -- 172.21.15.107:0/1920704216 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd0075030 con 0x7fecc81633a0 2023-12-04T05:40:07.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7ff012ffd640 1 --2- 172.21.15.107:0/285567384 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff00c0e6c00 0x7ff00c174a00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff01407b900 tx=0x7ff014068290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7fecbeffd640 1 -- 172.21.15.107:0/1920704216 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd0075190 con 0x7fecc81633a0 2023-12-04T05:40:07.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7fecbeffd640 1 -- 172.21.15.107:0/1920704216 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd006a7b0 con 0x7fecc81633a0 2023-12-04T05:40:07.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7f3de0ff9640 1 -- 172.21.15.107:0/4268987116 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0011830 con 0x7f3ddc0e71c0 2023-12-04T05:40:07.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7f489f59b640 1 -- 172.21.15.107:0/685017628 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4878005260 con 0x7f489806b4f0 2023-12-04T05:40:07.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7feccf59e640 1 -- 172.21.15.107:0/1920704216 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc80d7fc0 con 0x7fecc81633a0 2023-12-04T05:40:07.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7feccf59e640 1 -- 172.21.15.107:0/1920704216 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc8162e80 con 0x7fecc81633a0 2023-12-04T05:40:07.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7f3de0ff9640 1 -- 172.21.15.107:0/4268987116 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd0011990 con 0x7f3ddc0e71c0 2023-12-04T05:40:07.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7f3de0ff9640 1 -- 172.21.15.107:0/4268987116 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0011d40 con 0x7f3ddc0e71c0 2023-12-04T05:40:07.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7f3de957b640 1 -- 172.21.15.107:0/4268987116 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ddc178d10 con 0x7f3ddc0e71c0 2023-12-04T05:40:07.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7f3de957b640 1 -- 172.21.15.107:0/4268987116 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ddc1792d0 con 0x7f3ddc0e71c0 2023-12-04T05:40:07.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7f3de957b640 1 -- 172.21.15.107:0/4268987116 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ddc008910 con 0x7f3ddc0e71c0 2023-12-04T05:40:07.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7feccf59e640 1 -- 172.21.15.107:0/1920704216 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecc80057c0 con 0x7fecc81633a0 2023-12-04T05:40:07.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7f87aeffd640 1 -- 172.21.15.107:0/2085076697 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f87a80113f0 con 0x7f87b00dd290 2023-12-04T05:40:07.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7efdf77fe640 1 -- 172.21.15.107:0/3843017370 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe04010870 con 0x7efe100c7480 2023-12-04T05:40:07.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7efdf77fe640 1 -- 172.21.15.107:0/3843017370 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe040109f0 con 0x7efe100c7480 2023-12-04T05:40:07.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.707+0000 7efdf77fe640 1 -- 172.21.15.107:0/3843017370 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe04010da0 con 0x7efe100c7480 2023-12-04T05:40:07.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7efe0ffff640 1 -- 172.21.15.107:0/3843017370 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe101574e0 con 0x7efe100c7480 2023-12-04T05:40:07.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7efe0ffff640 1 -- 172.21.15.107:0/3843017370 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe10157aa0 con 0x7efe100c7480 2023-12-04T05:40:07.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7f11197fa640 1 -- 172.21.15.107:0/468814680 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f11100b7070 con 0x7f1114004ae0 2023-12-04T05:40:07.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7f87aeffd640 1 --2- 172.21.15.107:0/2085076697 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f879c0668d0 0x7f879c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7f87aeffd640 1 -- 172.21.15.107:0/2085076697 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87a8084540 con 0x7f87b00dd290 2023-12-04T05:40:07.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7f87bdc95640 1 --2- 172.21.15.107:0/2085076697 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f879c0668d0 0x7f879c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7f11197fa640 1 --2- 172.21.15.107:0/468814680 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f110c066b60 0x7f110c069020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7efe0ffff640 1 -- 172.21.15.107:0/3843017370 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe10157f80 con 0x7efe100c7480 2023-12-04T05:40:07.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7feffffff640 1 -- 172.21.15.107:0/285567384 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff014074e20 con 0x7ff00c0e6c00 2023-12-04T05:40:07.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7feffffff640 1 -- 172.21.15.107:0/285567384 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff014074f80 con 0x7ff00c0e6c00 2023-12-04T05:40:07.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7feffffff640 1 -- 172.21.15.107:0/285567384 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01406b730 con 0x7ff00c0e6c00 2023-12-04T05:40:07.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7ff018c7f640 1 -- 172.21.15.107:0/285567384 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff00c0d9440 con 0x7ff00c0e6c00 2023-12-04T05:40:07.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7ff018c7f640 1 -- 172.21.15.107:0/285567384 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff00c1823a0 con 0x7ff00c0e6c00 2023-12-04T05:40:07.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7f11197fa640 1 -- 172.21.15.107:0/468814680 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1110121fe0 con 0x7f1114004ae0 2023-12-04T05:40:07.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7f111bfff640 1 --2- 172.21.15.107:0/468814680 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f110c066b60 0x7f110c069020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7f111bfff640 1 --2- 172.21.15.107:0/468814680 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f110c066b60 0x7f110c069020 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f11140dc490 tx=0x7f1108009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7fecbeffd640 1 -- 172.21.15.107:0/1920704216 <== mon.2 v2:172.21.15.195:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fecd0098070 con 0x7fecc81633a0 2023-12-04T05:40:07.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7fecbeffd640 1 --2- 172.21.15.107:0/1920704216 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fecb00668d0 0x7fecb0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7fecce59c640 1 --2- 172.21.15.107:0/1920704216 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fecb00668d0 0x7fecb0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7fecbeffd640 1 -- 172.21.15.107:0/1920704216 <== mon.2 v2:172.21.15.195:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fecd0103890 con 0x7fecc81633a0 2023-12-04T05:40:07.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7fecce59c640 1 --2- 172.21.15.107:0/1920704216 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fecb00668d0 0x7fecb0068d90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fecc00ae000 tx=0x7fecc00aa070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7f3de0ff9640 1 -- 172.21.15.107:0/4268987116 <== mon.2 v2:172.21.15.195:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f3dd001a400 con 0x7f3ddc0e71c0 2023-12-04T05:40:07.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7f3de0ff9640 1 --2- 172.21.15.107:0/4268987116 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3dc8066920 0x7f3dc8068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7f3de0ff9640 1 -- 172.21.15.107:0/4268987116 <== mon.2 v2:172.21.15.195:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3dd0084820 con 0x7f3ddc0e71c0 2023-12-04T05:40:07.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7f3de3fff640 1 --2- 172.21.15.107:0/4268987116 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3dc8066920 0x7f3dc8068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7f3de3fff640 1 --2- 172.21.15.107:0/4268987116 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3dc8066920 0x7f3dc8068de0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc00ddd0 tx=0x7f3de4125000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.711+0000 7ff018c7f640 1 -- 172.21.15.107:0/285567384 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff4005260 con 0x7ff00c0e6c00 2023-12-04T05:40:07.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.715+0000 7f87c06fa640 1 -- 172.21.15.107:0/2085076697 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b017c140 con 0x7f87b00dd290 2023-12-04T05:40:07.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.715+0000 7f87bdc95640 1 --2- 172.21.15.107:0/2085076697 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f879c0668d0 0x7f879c068d90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f87b0179b20 tx=0x7f87b4006c50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.715+0000 7f11197fa640 1 -- 172.21.15.107:0/468814680 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f11100ef340 con 0x7f1114004ae0 2023-12-04T05:40:07.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.715+0000 7ffb5e7fc640 1 -- 172.21.15.107:0/3241084504 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb58011830 con 0x7ffb6814ffb0 2023-12-04T05:40:07.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.715+0000 7ffb6fd54640 1 -- 172.21.15.107:0/3241084504 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb6814b240 con 0x7ffb6814ffb0 2023-12-04T05:40:07.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.715+0000 7ffb6fd54640 1 -- 172.21.15.107:0/3241084504 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb681f2120 con 0x7ffb6814ffb0 2023-12-04T05:40:07.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.715+0000 7ffb6fd54640 1 -- 172.21.15.107:0/3241084504 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb68068770 con 0x7ffb6814ffb0 2023-12-04T05:40:07.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.715+0000 7f488dffb640 1 -- 172.21.15.107:0/685017628 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4894013470 con 0x7f489806b4f0 2023-12-04T05:40:07.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.715+0000 7f488dffb640 1 --2- 172.21.15.107:0/685017628 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4884066a60 0x7f4884068f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.715+0000 7f488dffb640 1 -- 172.21.15.107:0/685017628 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4894084860 con 0x7f489806b4f0 2023-12-04T05:40:07.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.715+0000 7f489d337640 1 --2- 172.21.15.107:0/685017628 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4884066a60 0x7f4884068f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.715+0000 7ffb5e7fc640 1 -- 172.21.15.107:0/3241084504 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb58011990 con 0x7ffb6814ffb0 2023-12-04T05:40:07.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.715+0000 7ffb5e7fc640 1 -- 172.21.15.107:0/3241084504 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb58011d20 con 0x7ffb6814ffb0 2023-12-04T05:40:07.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.719+0000 7efdf77fe640 1 -- 172.21.15.107:0/3843017370 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7efe04023480 con 0x7efe100c7480 2023-12-04T05:40:07.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.719+0000 7efdf77fe640 1 --2- 172.21.15.107:0/3843017370 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efdfc066950 0x7efdfc068e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.719+0000 7efdf77fe640 1 -- 172.21.15.107:0/3843017370 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efe04084840 con 0x7efe100c7480 2023-12-04T05:40:07.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.719+0000 7f489d337640 1 --2- 172.21.15.107:0/685017628 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4884066a60 0x7f4884068f20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f48880b2000 tx=0x7f4888008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.719+0000 7efe0effd640 1 --2- 172.21.15.107:0/3843017370 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efdfc066950 0x7efdfc068e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.719+0000 7efe0effd640 1 --2- 172.21.15.107:0/3843017370 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efdfc066950 0x7efdfc068e10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efe000b0000 tx=0x7efe000a8070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.723+0000 7f488dffb640 1 -- 172.21.15.107:0/685017628 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4894051c30 con 0x7f489806b4f0 2023-12-04T05:40:07.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.727+0000 7fecbeffd640 1 -- 172.21.15.107:0/1920704216 <== mon.2 v2:172.21.15.195:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fecd00d0a20 con 0x7fecc81633a0 2023-12-04T05:40:07.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.727+0000 7ffb5e7fc640 1 -- 172.21.15.107:0/3241084504 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ffb5801a400 con 0x7ffb6814ffb0 2023-12-04T05:40:07.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.727+0000 7ffb5e7fc640 1 --2- 172.21.15.107:0/3241084504 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ffb4c0668d0 0x7ffb4c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.727+0000 7ffb6daf0640 1 --2- 172.21.15.107:0/3241084504 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ffb4c0668d0 0x7ffb4c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.727+0000 7ffb6daf0640 1 --2- 172.21.15.107:0/3241084504 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ffb4c0668d0 0x7ffb4c068d90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ffb6400b240 tx=0x7ffb6400b3a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.731+0000 7efdf77fe640 1 -- 172.21.15.107:0/3843017370 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7efe04051b10 con 0x7efe100c7480 2023-12-04T05:40:07.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.731+0000 7ffb5e7fc640 1 -- 172.21.15.107:0/3241084504 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb580559c0 con 0x7ffb6814ffb0 2023-12-04T05:40:07.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.731+0000 7f87aeffd640 1 -- 172.21.15.107:0/2085076697 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f87b017c140 con 0x7f87b00dd290 2023-12-04T05:40:07.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.735+0000 7ffb5e7fc640 1 -- 172.21.15.107:0/3241084504 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ffb58051d00 con 0x7ffb6814ffb0 2023-12-04T05:40:07.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.739+0000 7f3de0ff9640 1 -- 172.21.15.107:0/4268987116 <== mon.2 v2:172.21.15.195:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3dd0051b70 con 0x7f3ddc0e71c0 2023-12-04T05:40:07.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.743+0000 7feffffff640 1 -- 172.21.15.107:0/285567384 <== mon.2 v2:172.21.15.195:3301/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff01406b890 con 0x7ff00c0e6c00 2023-12-04T05:40:07.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.743+0000 7feffffff640 1 --2- 172.21.15.107:0/285567384 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7feff80669d0 0x7feff8068e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:07.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.743+0000 7feffffff640 1 -- 172.21.15.107:0/285567384 <== mon.2 v2:172.21.15.195:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0140fefe0 con 0x7ff00c0e6c00 2023-12-04T05:40:07.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.743+0000 7ff0137fe640 1 --2- 172.21.15.107:0/285567384 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7feff80669d0 0x7feff8068e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:07.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.743+0000 7ff0137fe640 1 --2- 172.21.15.107:0/285567384 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7feff80669d0 0x7feff8068e90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff00c00db40 tx=0x7ff0080a6000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:07.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:07.755+0000 7feffffff640 1 -- 172.21.15.107:0/285567384 <== mon.2 v2:172.21.15.195:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff0140cc290 con 0x7ff00c0e6c00 2023-12-04T05:40:08.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.015+0000 7f1122b5c640 1 -- 172.21.15.107:0/468814680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f111404bb40 con 0x7f1114004ae0 2023-12-04T05:40:08.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.019+0000 7f11197fa640 1 -- 172.21.15.107:0/468814680 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f11100c1020 con 0x7f1114004ae0 2023-12-04T05:40:08.022 INFO:teuthology.orchestra.run.smithi107.stdout:55834574854 2023-12-04T05:40:08.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.023+0000 7f1106ffd640 1 -- 172.21.15.107:0/468814680 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f110c066b60 msgr2=0x7f110c069020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.023+0000 7f1106ffd640 1 --2- 172.21.15.107:0/468814680 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f110c066b60 0x7f110c069020 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f11140dc490 tx=0x7f1108009040 comp rx=0 tx=0).stop 2023-12-04T05:40:08.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.023+0000 7f1106ffd640 1 -- 172.21.15.107:0/468814680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1114004ae0 msgr2=0x7f11140d9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.023+0000 7f1106ffd640 1 --2- 172.21.15.107:0/468814680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1114004ae0 0x7f11140d9e00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f11100b3040 tx=0x7f111001b9d0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.023+0000 7f1106ffd640 1 -- 172.21.15.107:0/468814680 shutdown_connections 2023-12-04T05:40:08.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.023+0000 7f1106ffd640 1 --2- 172.21.15.107:0/468814680 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f110c066b60 0x7f110c069020 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.023+0000 7f1106ffd640 1 --2- 172.21.15.107:0/468814680 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f11140d6cc0 0x7f11140cd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.023+0000 7f1106ffd640 1 --2- 172.21.15.107:0/468814680 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f111404c620 0x7f11140db970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.023+0000 7f1106ffd640 1 --2- 172.21.15.107:0/468814680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1114004ae0 0x7f11140d9e00 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.023+0000 7f1106ffd640 1 -- 172.21.15.107:0/468814680 >> 172.21.15.107:0/468814680 conn(0x7f11140505b0 msgr2=0x7f111404ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:08.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.023+0000 7f1106ffd640 1 -- 172.21.15.107:0/468814680 shutdown_connections 2023-12-04T05:40:08.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.023+0000 7f1106ffd640 1 -- 172.21.15.107:0/468814680 wait complete. 2023-12-04T05:40:08.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.031+0000 7feccf59e640 1 -- 172.21.15.107:0/1920704216 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fecc804bb40 con 0x7fecc81633a0 2023-12-04T05:40:08.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.031+0000 7fecbeffd640 1 -- 172.21.15.107:0/1920704216 <== mon.2 v2:172.21.15.195:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fecd00d4040 con 0x7fecc81633a0 2023-12-04T05:40:08.034 INFO:teuthology.orchestra.run.smithi107.stdout:55834574855 2023-12-04T05:40:08.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.031+0000 7f489f59b640 1 -- 172.21.15.107:0/685017628 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f48780057e0 con 0x7f489806b4f0 2023-12-04T05:40:08.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.031+0000 7f488dffb640 1 -- 172.21.15.107:0/685017628 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4894055250 con 0x7f489806b4f0 2023-12-04T05:40:08.035 INFO:teuthology.orchestra.run.smithi107.stdout:55834574854 2023-12-04T05:40:08.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.031+0000 7f3de957b640 1 -- 172.21.15.107:0/4268987116 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f3ddc008910 con 0x7f3ddc0e71c0 2023-12-04T05:40:08.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.031+0000 7f3de0ff9640 1 -- 172.21.15.107:0/4268987116 <== mon.2 v2:172.21.15.195:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3ddc008910 con 0x7f3ddc0e71c0 2023-12-04T05:40:08.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7feccf59e640 1 -- 172.21.15.107:0/1920704216 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fecb00668d0 msgr2=0x7fecb0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7feccf59e640 1 --2- 172.21.15.107:0/1920704216 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fecb00668d0 0x7fecb0068d90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fecc00ae000 tx=0x7fecc00aa070 comp rx=0 tx=0).stop 2023-12-04T05:40:08.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7feccf59e640 1 -- 172.21.15.107:0/1920704216 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fecc81633a0 msgr2=0x7fecc8163820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7feccf59e640 1 --2- 172.21.15.107:0/1920704216 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fecc81633a0 0x7fecc8163820 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fecd004fba0 tx=0x7fecd0062130 comp rx=0 tx=0).stop 2023-12-04T05:40:08.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7feccf59e640 1 -- 172.21.15.107:0/1920704216 shutdown_connections 2023-12-04T05:40:08.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7feccf59e640 1 --2- 172.21.15.107:0/1920704216 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fecb00668d0 0x7fecb0068d90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7feccf59e640 1 --2- 172.21.15.107:0/1920704216 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fecc81633a0 0x7fecc8163820 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7feccf59e640 1 --2- 172.21.15.107:0/1920704216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecc8164e10 0x7fecc8005ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7feccf59e640 1 --2- 172.21.15.107:0/1920704216 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fecb8003b10 0x7fecc8172130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7feccf59e640 1 -- 172.21.15.107:0/1920704216 >> 172.21.15.107:0/1920704216 conn(0x7fecc80505b0 msgr2=0x7fecc8050990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:08.038 INFO:teuthology.orchestra.run.smithi107.stdout:55834574856 2023-12-04T05:40:08.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f489f59b640 1 -- 172.21.15.107:0/685017628 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4884066a60 msgr2=0x7f4884068f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f489f59b640 1 --2- 172.21.15.107:0/685017628 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4884066a60 0x7f4884068f20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f48880b2000 tx=0x7f4888008040 comp rx=0 tx=0).stop 2023-12-04T05:40:08.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f489f59b640 1 -- 172.21.15.107:0/685017628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f489806b4f0 msgr2=0x7f48980bce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f489f59b640 1 --2- 172.21.15.107:0/685017628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f489806b4f0 0x7f48980bce70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f489400e870 tx=0x7f489400ed30 comp rx=0 tx=0).stop 2023-12-04T05:40:08.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7feccf59e640 1 -- 172.21.15.107:0/1920704216 shutdown_connections 2023-12-04T05:40:08.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f489f59b640 1 -- 172.21.15.107:0/685017628 shutdown_connections 2023-12-04T05:40:08.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f489f59b640 1 --2- 172.21.15.107:0/685017628 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4884066a60 0x7f4884068f20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f489f59b640 1 --2- 172.21.15.107:0/685017628 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48980c5ba0 0x7f48980bd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f489f59b640 1 --2- 172.21.15.107:0/685017628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f489806b4f0 0x7f48980bce70 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f489f59b640 1 --2- 172.21.15.107:0/685017628 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f489806a5b0 0x7f489806d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f489f59b640 1 -- 172.21.15.107:0/685017628 >> 172.21.15.107:0/685017628 conn(0x7f48980cb2a0 msgr2=0x7f48980707e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:08.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f489f59b640 1 -- 172.21.15.107:0/685017628 shutdown_connections 2023-12-04T05:40:08.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f489f59b640 1 -- 172.21.15.107:0/685017628 wait complete. 2023-12-04T05:40:08.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7feccf59e640 1 -- 172.21.15.107:0/1920704216 wait complete. 2023-12-04T05:40:08.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f3de957b640 1 -- 172.21.15.107:0/4268987116 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3dc8066920 msgr2=0x7f3dc8068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f3de957b640 1 --2- 172.21.15.107:0/4268987116 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3dc8066920 0x7f3dc8068de0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc00ddd0 tx=0x7f3de4125000 comp rx=0 tx=0).stop 2023-12-04T05:40:08.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f3de957b640 1 -- 172.21.15.107:0/4268987116 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3ddc0e71c0 msgr2=0x7f3ddc0d5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f3de957b640 1 --2- 172.21.15.107:0/4268987116 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3ddc0e71c0 0x7f3ddc0d5fc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3dd000b410 tx=0x7f3dd000b8d0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f3de957b640 1 -- 172.21.15.107:0/4268987116 shutdown_connections 2023-12-04T05:40:08.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f3de957b640 1 --2- 172.21.15.107:0/4268987116 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3dc8066920 0x7f3dc8068de0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f3de957b640 1 --2- 172.21.15.107:0/4268987116 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3ddc0e71c0 0x7f3ddc0d5fc0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f3de957b640 1 --2- 172.21.15.107:0/4268987116 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3ddc0dd290 0x7f3ddc17c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f3de957b640 1 --2- 172.21.15.107:0/4268987116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ddc046270 0x7f3ddc16fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f3de957b640 1 -- 172.21.15.107:0/4268987116 >> 172.21.15.107:0/4268987116 conn(0x7f3ddc050160 msgr2=0x7f3ddc04f210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:08.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f3de957b640 1 -- 172.21.15.107:0/4268987116 shutdown_connections 2023-12-04T05:40:08.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.035+0000 7f3de957b640 1 -- 172.21.15.107:0/4268987116 wait complete. 2023-12-04T05:40:08.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.039+0000 7efe0ffff640 1 -- 172.21.15.107:0/3843017370 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7efe1004e890 con 0x7efe100c7480 2023-12-04T05:40:08.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.043+0000 7efdf77fe640 1 -- 172.21.15.107:0/3843017370 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efe04082e60 con 0x7efe100c7480 2023-12-04T05:40:08.046 INFO:teuthology.orchestra.run.smithi107.stdout:55834574856 2023-12-04T05:40:08.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.043+0000 7efdf57fa640 1 -- 172.21.15.107:0/3843017370 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efdfc066950 msgr2=0x7efdfc068e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.043+0000 7efdf57fa640 1 --2- 172.21.15.107:0/3843017370 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efdfc066950 0x7efdfc068e10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efe000b0000 tx=0x7efe000a8070 comp rx=0 tx=0).stop 2023-12-04T05:40:08.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.047+0000 7efdf57fa640 1 -- 172.21.15.107:0/3843017370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe100c7480 msgr2=0x7efe101598c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.047+0000 7efdf57fa640 1 --2- 172.21.15.107:0/3843017370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe100c7480 0x7efe101598c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efe0400b480 tx=0x7efe0400b940 comp rx=0 tx=0).stop 2023-12-04T05:40:08.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.047+0000 7efdf57fa640 1 -- 172.21.15.107:0/3843017370 shutdown_connections 2023-12-04T05:40:08.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.047+0000 7efdf57fa640 1 --2- 172.21.15.107:0/3843017370 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efdfc066950 0x7efdfc068e10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.047+0000 7efdf57fa640 1 --2- 172.21.15.107:0/3843017370 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efe1014c6c0 0x7efe10149f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.047+0000 7efdf57fa640 1 --2- 172.21.15.107:0/3843017370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe100c7480 0x7efe101598c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.047+0000 7efdf57fa640 1 --2- 172.21.15.107:0/3843017370 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efe1006ff80 0x7efe101ff780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.047+0000 7efdf57fa640 1 -- 172.21.15.107:0/3843017370 >> 172.21.15.107:0/3843017370 conn(0x7efe100cb3a0 msgr2=0x7efe100bdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:08.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.047+0000 7efdf57fa640 1 -- 172.21.15.107:0/3843017370 shutdown_connections 2023-12-04T05:40:08.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.047+0000 7efdf57fa640 1 -- 172.21.15.107:0/3843017370 wait complete. 2023-12-04T05:40:08.052 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2023-12-04T05:40:08.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.047+0000 7ffb6fd54640 1 -- 172.21.15.107:0/3241084504 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7ffb681f2640 con 0x7ffb6814ffb0 2023-12-04T05:40:08.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.047+0000 7ffb5e7fc640 1 -- 172.21.15.107:0/3241084504 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ffb58015020 con 0x7ffb6814ffb0 2023-12-04T05:40:08.053 DEBUG:teuthology.parallel:result is None 2023-12-04T05:40:08.053 INFO:teuthology.orchestra.run.smithi107.stdout:55834574856 2023-12-04T05:40:08.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.051+0000 7f87c06fa640 1 -- 172.21.15.107:0/2085076697 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f87b017c140 con 0x7f87b00dd290 2023-12-04T05:40:08.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.051+0000 7ffb6fd54640 1 -- 172.21.15.107:0/3241084504 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ffb4c0668d0 msgr2=0x7ffb4c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.051+0000 7ffb6fd54640 1 --2- 172.21.15.107:0/3241084504 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ffb4c0668d0 0x7ffb4c068d90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ffb6400b240 tx=0x7ffb6400b3a0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.051+0000 7ffb6fd54640 1 -- 172.21.15.107:0/3241084504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb6814ffb0 msgr2=0x7ffb681f2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.051+0000 7ffb6fd54640 1 --2- 172.21.15.107:0/3241084504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb6814ffb0 0x7ffb681f2920 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffb5800b3e0 tx=0x7ffb5800b8a0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.055+0000 7ffb6fd54640 1 -- 172.21.15.107:0/3241084504 shutdown_connections 2023-12-04T05:40:08.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.055+0000 7ffb6fd54640 1 --2- 172.21.15.107:0/3241084504 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ffb4c0668d0 0x7ffb4c068d90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.055+0000 7ffb6fd54640 1 --2- 172.21.15.107:0/3241084504 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffb68150390 0x7ffb680c9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.055+0000 7ffb6fd54640 1 --2- 172.21.15.107:0/3241084504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb6814ffb0 0x7ffb681f2920 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.055+0000 7ffb6fd54640 1 --2- 172.21.15.107:0/3241084504 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffb680c77d0 0x7ffb6814b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.055+0000 7ffb6fd54640 1 -- 172.21.15.107:0/3241084504 >> 172.21.15.107:0/3241084504 conn(0x7ffb680cb2a0 msgr2=0x7ffb68155920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:08.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.055+0000 7f87aeffd640 1 -- 172.21.15.107:0/2085076697 <== mon.1 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f87b017c140 con 0x7f87b00dd290 2023-12-04T05:40:08.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.055+0000 7ffb6fd54640 1 -- 172.21.15.107:0/3241084504 shutdown_connections 2023-12-04T05:40:08.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.055+0000 7ffb6fd54640 1 -- 172.21.15.107:0/3241084504 wait complete. 2023-12-04T05:40:08.063 INFO:teuthology.orchestra.run.smithi107.stdout:55834574854 2023-12-04T05:40:08.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.063+0000 7f87c06fa640 1 -- 172.21.15.107:0/2085076697 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f879c0668d0 msgr2=0x7f879c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.063+0000 7f87c06fa640 1 --2- 172.21.15.107:0/2085076697 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f879c0668d0 0x7f879c068d90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f87b0179b20 tx=0x7f87b4006c50 comp rx=0 tx=0).stop 2023-12-04T05:40:08.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.063+0000 7f87c06fa640 1 -- 172.21.15.107:0/2085076697 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87b00dd290 msgr2=0x7f87b00499e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.063+0000 7f87c06fa640 1 --2- 172.21.15.107:0/2085076697 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87b00dd290 0x7f87b00499e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f87a800c7f0 tx=0x7f87a800ccb0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.063+0000 7f87c06fa640 1 -- 172.21.15.107:0/2085076697 shutdown_connections 2023-12-04T05:40:08.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.063+0000 7f87c06fa640 1 --2- 172.21.15.107:0/2085076697 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f879c0668d0 0x7f879c068d90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.063+0000 7f87c06fa640 1 --2- 172.21.15.107:0/2085076697 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f87b00e71c0 0x7f87b01789f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.063+0000 7f87c06fa640 1 --2- 172.21.15.107:0/2085076697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87b00e6c00 0x7f87b00dad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.063+0000 7f87c06fa640 1 --2- 172.21.15.107:0/2085076697 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87b00dd290 0x7f87b00499e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.063+0000 7f87c06fa640 1 -- 172.21.15.107:0/2085076697 >> 172.21.15.107:0/2085076697 conn(0x7f87b0050160 msgr2=0x7f87b0050540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:08.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.063+0000 7f87c06fa640 1 -- 172.21.15.107:0/2085076697 shutdown_connections 2023-12-04T05:40:08.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.063+0000 7f87c06fa640 1 -- 172.21.15.107:0/2085076697 wait complete. 2023-12-04T05:40:08.069 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.3 2023-12-04T05:40:08.069 DEBUG:teuthology.parallel:result is None 2023-12-04T05:40:08.072 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.6 2023-12-04T05:40:08.072 DEBUG:teuthology.parallel:result is None 2023-12-04T05:40:08.073 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.4 2023-12-04T05:40:08.074 DEBUG:teuthology.parallel:result is None 2023-12-04T05:40:08.075 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.1 2023-12-04T05:40:08.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.071+0000 7ff018c7f640 1 -- 172.21.15.107:0/285567384 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7feff40050d0 con 0x7ff00c0e6c00 2023-12-04T05:40:08.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.071+0000 7feffffff640 1 -- 172.21.15.107:0/285567384 <== mon.2 v2:172.21.15.195:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff014090090 con 0x7ff00c0e6c00 2023-12-04T05:40:08.075 DEBUG:teuthology.parallel:result is None 2023-12-04T05:40:08.076 INFO:teuthology.orchestra.run.smithi107.stdout:55834574854 2023-12-04T05:40:08.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.075+0000 7ff018c7f640 1 -- 172.21.15.107:0/285567384 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7feff80669d0 msgr2=0x7feff8068e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.075+0000 7ff018c7f640 1 --2- 172.21.15.107:0/285567384 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7feff80669d0 0x7feff8068e90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff00c00db40 tx=0x7ff0080a6000 comp rx=0 tx=0).stop 2023-12-04T05:40:08.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.075+0000 7ff018c7f640 1 -- 172.21.15.107:0/285567384 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff00c0e6c00 msgr2=0x7ff00c174a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.075+0000 7ff018c7f640 1 --2- 172.21.15.107:0/285567384 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff00c0e6c00 0x7ff00c174a00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff01407b900 tx=0x7ff014068290 comp rx=0 tx=0).stop 2023-12-04T05:40:08.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.075+0000 7ff018c7f640 1 -- 172.21.15.107:0/285567384 shutdown_connections 2023-12-04T05:40:08.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.075+0000 7ff018c7f640 1 --2- 172.21.15.107:0/285567384 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7feff80669d0 0x7feff8068e90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.075+0000 7ff018c7f640 1 --2- 172.21.15.107:0/285567384 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff00c0e71c0 0x7ff00c0d5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.075+0000 7ff018c7f640 1 --2- 172.21.15.107:0/285567384 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff00c0e6c00 0x7ff00c174a00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.075+0000 7ff018c7f640 1 --2- 172.21.15.107:0/285567384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff00c0dd290 0x7ff00c1744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.075+0000 7ff018c7f640 1 -- 172.21.15.107:0/285567384 >> 172.21.15.107:0/285567384 conn(0x7ff00c050160 msgr2=0x7ff00c006590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:08.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.075+0000 7ff018c7f640 1 -- 172.21.15.107:0/285567384 shutdown_connections 2023-12-04T05:40:08.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.075+0000 7ff018c7f640 1 -- 172.21.15.107:0/285567384 wait complete. 2023-12-04T05:40:08.084 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.7 2023-12-04T05:40:08.084 DEBUG:teuthology.parallel:result is None 2023-12-04T05:40:08.087 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2023-12-04T05:40:08.087 DEBUG:teuthology.parallel:result is None 2023-12-04T05:40:08.095 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2023-12-04T05:40:08.095 DEBUG:teuthology.parallel:result is None 2023-12-04T05:40:08.095 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-04T05:40:08.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-04T05:40:08.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.235+0000 7efceb3fc640 1 Processor -- start 2023-12-04T05:40:08.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.235+0000 7efceb3fc640 1 -- start start 2023-12-04T05:40:08.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efceb3fc640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efce4154020 0x7efce415a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:08.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efceb3fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efce40d01c0 0x7efce4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:08.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efceb3fc640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7efce4072220 0x7efce40c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:08.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efceb3fc640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efce40ca230 con 0x7efce40d01c0 2023-12-04T05:40:08.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efceb3fc640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efce40c6770 con 0x7efce4072220 2023-12-04T05:40:08.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efceb3fc640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce415ad50 con 0x7efce4154020 2023-12-04T05:40:08.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efce8997640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efce40d01c0 0x7efce4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:08.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efce8997640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efce40d01c0 0x7efce4071ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40912/0 (socket says 172.21.15.107:40912) 2023-12-04T05:40:08.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efce8997640 1 -- 172.21.15.107:0/1655118021 learned_addr learned my addr 172.21.15.107:0/1655118021 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:08.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efce9999640 1 --2- 172.21.15.107:0/1655118021 >> v2:172.21.15.195:3300/0 conn(0x7efce4072220 0x7efce40c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:08.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efce9198640 1 --2- 172.21.15.107:0/1655118021 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efce4154020 0x7efce415a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:08.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efce8997640 1 -- 172.21.15.107:0/1655118021 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efce4154020 msgr2=0x7efce415a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:08.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efce8997640 1 --2- 172.21.15.107:0/1655118021 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efce4154020 0x7efce415a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efce8997640 1 -- 172.21.15.107:0/1655118021 >> v2:172.21.15.195:3300/0 conn(0x7efce4072220 msgr2=0x7efce40c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:08.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efce8997640 1 --2- 172.21.15.107:0/1655118021 >> v2:172.21.15.195:3300/0 conn(0x7efce4072220 0x7efce40c4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efce8997640 1 -- 172.21.15.107:0/1655118021 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce415d710 con 0x7efce40d01c0 2023-12-04T05:40:08.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efce9999640 1 --2- 172.21.15.107:0/1655118021 >> v2:172.21.15.195:3300/0 conn(0x7efce4072220 0x7efce40c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:08.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efce8997640 1 --2- 172.21.15.107:0/1655118021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce40d01c0 0x7efce4071ce0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efce40caa70 tx=0x7efcd401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b2569ef7e068b3a server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:08.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efcdbfff640 1 -- 172.21.15.107:0/1655118021 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd40a89b0 con 0x7efce40d01c0 2023-12-04T05:40:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efcdbfff640 1 -- 172.21.15.107:0/1655118021 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd40a8b10 con 0x7efce40d01c0 2023-12-04T05:40:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efcdbfff640 1 -- 172.21.15.107:0/1655118021 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd40a8ea0 con 0x7efce40d01c0 2023-12-04T05:40:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efceb3fc640 1 -- 172.21.15.107:0/1655118021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce40d01c0 msgr2=0x7efce4071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efceb3fc640 1 --2- 172.21.15.107:0/1655118021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce40d01c0 0x7efce4071ce0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efce40caa70 tx=0x7efcd401ecf0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efceb3fc640 1 -- 172.21.15.107:0/1655118021 shutdown_connections 2023-12-04T05:40:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efceb3fc640 1 --2- 172.21.15.107:0/1655118021 >> v2:172.21.15.195:3300/0 conn(0x7efce4072220 0x7efce40c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efceb3fc640 1 --2- 172.21.15.107:0/1655118021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce40d01c0 0x7efce4071ce0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efceb3fc640 1 --2- 172.21.15.107:0/1655118021 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efce4154020 0x7efce415a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efceb3fc640 1 -- 172.21.15.107:0/1655118021 >> 172.21.15.107:0/1655118021 conn(0x7efce406c6c0 msgr2=0x7efce40cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efceb3fc640 1 -- 172.21.15.107:0/1655118021 shutdown_connections 2023-12-04T05:40:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.239+0000 7efceb3fc640 1 -- 172.21.15.107:0/1655118021 wait complete. 2023-12-04T05:40:08.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efceb3fc640 1 Processor -- start 2023-12-04T05:40:08.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efceb3fc640 1 -- start start 2023-12-04T05:40:08.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efceb3fc640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efce4072220 0x7efce41ef8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:08.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efceb3fc640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efce40d01c0 0x7efce41585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:08.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efceb3fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce4154020 0x7efce40c8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:08.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efceb3fc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce41f3700 con 0x7efce4154020 2023-12-04T05:40:08.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efceb3fc640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce41f39e0 con 0x7efce40d01c0 2023-12-04T05:40:08.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efceb3fc640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce41f3cc0 con 0x7efce4072220 2023-12-04T05:40:08.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efce9999640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce4154020 0x7efce40c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:08.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efce9999640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce4154020 0x7efce40c8140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40916/0 (socket says 172.21.15.107:40916) 2023-12-04T05:40:08.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efce9999640 1 -- 172.21.15.107:0/1659801815 learned_addr learned my addr 172.21.15.107:0/1659801815 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:08.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efce9198640 1 --2- 172.21.15.107:0/1659801815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efce4072220 0x7efce41ef8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:08.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efce8997640 1 --2- 172.21.15.107:0/1659801815 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efce40d01c0 0x7efce41585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:08.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efce9999640 1 -- 172.21.15.107:0/1659801815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efce4072220 msgr2=0x7efce41ef8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efce9999640 1 --2- 172.21.15.107:0/1659801815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efce4072220 0x7efce41ef8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efce9999640 1 -- 172.21.15.107:0/1659801815 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efce40d01c0 msgr2=0x7efce41585d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efce9999640 1 --2- 172.21.15.107:0/1659801815 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efce40d01c0 0x7efce41585d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efce9999640 1 -- 172.21.15.107:0/1659801815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd40094d0 con 0x7efce4154020 2023-12-04T05:40:08.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efce8997640 1 --2- 172.21.15.107:0/1659801815 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efce40d01c0 0x7efce41585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:08.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efce9999640 1 --2- 172.21.15.107:0/1659801815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce4154020 0x7efce40c8140 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00cd10 tx=0x7efcdc00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:08.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efcd9ffb640 1 -- 172.21.15.107:0/1659801815 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc00f680 con 0x7efce4154020 2023-12-04T05:40:08.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efcd9ffb640 1 -- 172.21.15.107:0/1659801815 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcdc00f800 con 0x7efce4154020 2023-12-04T05:40:08.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efceb3fc640 1 -- 172.21.15.107:0/1659801815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce4157af0 con 0x7efce4154020 2023-12-04T05:40:08.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efcd9ffb640 1 -- 172.21.15.107:0/1659801815 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc00fbb0 con 0x7efce4154020 2023-12-04T05:40:08.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efceb3fc640 1 -- 172.21.15.107:0/1659801815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce41580b0 con 0x7efce4154020 2023-12-04T05:40:08.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.243+0000 7efceb3fc640 1 -- 172.21.15.107:0/1659801815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efce41582a0 con 0x7efce4154020 2023-12-04T05:40:08.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.247+0000 7efcd9ffb640 1 -- 172.21.15.107:0/1659801815 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7efcdc00b040 con 0x7efce4154020 2023-12-04T05:40:08.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.247+0000 7efcd9ffb640 1 --2- 172.21.15.107:0/1659801815 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efcc80668d0 0x7efcc8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:08.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.247+0000 7efcd9ffb640 1 -- 172.21.15.107:0/1659801815 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efcdc084e20 con 0x7efce4154020 2023-12-04T05:40:08.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.247+0000 7efce9198640 1 --2- 172.21.15.107:0/1659801815 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efcc80668d0 0x7efcc8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:08.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.247+0000 7efce9198640 1 --2- 172.21.15.107:0/1659801815 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efcc80668d0 0x7efcc8068d90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efce0009790 tx=0x7efce0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.251+0000 7efcd9ffb640 1 -- 172.21.15.107:0/1659801815 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7efcdc052060 con 0x7efce4154020 2023-12-04T05:40:08.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.387+0000 7efceb3fc640 1 -- 172.21.15.107:0/1659801815 --> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efce40c5ff0 con 0x7efcc80668d0 2023-12-04T05:40:08.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.395+0000 7efcd9ffb640 1 -- 172.21.15.107:0/1659801815 <== mgr.4104 v2:172.21.15.107:6832/2693048026 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264503 (secure 0 0 0) 0x7efce40c5ff0 con 0x7efcc80668d0 2023-12-04T05:40:08.401 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:08.402 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-12-04T05:40:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.399+0000 7efceb3fc640 1 -- 172.21.15.107:0/1659801815 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efcc80668d0 msgr2=0x7efcc8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.399+0000 7efceb3fc640 1 --2- 172.21.15.107:0/1659801815 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efcc80668d0 0x7efcc8068d90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efce0009790 tx=0x7efce0008040 comp rx=0 tx=0).stop 2023-12-04T05:40:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.399+0000 7efceb3fc640 1 -- 172.21.15.107:0/1659801815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce4154020 msgr2=0x7efce40c8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.399+0000 7efceb3fc640 1 --2- 172.21.15.107:0/1659801815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce4154020 0x7efce40c8140 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00cd10 tx=0x7efcdc00a650 comp rx=0 tx=0).stop 2023-12-04T05:40:08.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.399+0000 7efceb3fc640 1 -- 172.21.15.107:0/1659801815 shutdown_connections 2023-12-04T05:40:08.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.399+0000 7efceb3fc640 1 --2- 172.21.15.107:0/1659801815 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efcc80668d0 0x7efcc8068d90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.399+0000 7efceb3fc640 1 --2- 172.21.15.107:0/1659801815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce4154020 0x7efce40c8140 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.399+0000 7efceb3fc640 1 --2- 172.21.15.107:0/1659801815 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efce40d01c0 0x7efce41585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.399+0000 7efceb3fc640 1 --2- 172.21.15.107:0/1659801815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efce4072220 0x7efce41ef8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.399+0000 7efceb3fc640 1 -- 172.21.15.107:0/1659801815 >> 172.21.15.107:0/1659801815 conn(0x7efce406c6c0 msgr2=0x7efce4152550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:08.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.399+0000 7efceb3fc640 1 -- 172.21.15.107:0/1659801815 shutdown_connections 2023-12-04T05:40:08.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.399+0000 7efceb3fc640 1 -- 172.21.15.107:0/1659801815 wait complete. 2023-12-04T05:40:08.417 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-12-04T05:40:07.493008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235688,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739032,"statfs":{"total":773094113280,"available":772852768768,"internally_reserved":0,"allocated":2990080,"data_stored":1790204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002604"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503577+0000","last_change":"2023-12-04T05:39:59.736570+0000","last_active":"2023-12-04T05:40:01.503577+0000","last_peered":"2023-12-04T05:40:01.503577+0000","last_clean":"2023-12-04T05:40:01.503577+0000","last_became_active":"2023-12-04T05:39:59.736000+0000","last_became_peered":"2023-12-04T05:39:59.736000+0000","last_unstale":"2023-12-04T05:40:01.503577+0000","last_undegraded":"2023-12-04T05:40:01.503577+0000","last_fullsized":"2023-12-04T05:40:01.503577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:34:15.458889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908781+0000","last_change":"2023-12-04T05:39:59.703955+0000","last_active":"2023-12-04T05:40:00.908781+0000","last_peered":"2023-12-04T05:40:00.908781+0000","last_clean":"2023-12-04T05:40:00.908781+0000","last_became_active":"2023-12-04T05:39:59.703032+0000","last_became_peered":"2023-12-04T05:39:59.703032+0000","last_unstale":"2023-12-04T05:40:00.908781+0000","last_undegraded":"2023-12-04T05:40:00.908781+0000","last_fullsized":"2023-12-04T05:40:00.908781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:22:09.010348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501353+0000","last_change":"2023-12-04T05:39:59.705340+0000","last_active":"2023-12-04T05:40:01.501353+0000","last_peered":"2023-12-04T05:40:01.501353+0000","last_clean":"2023-12-04T05:40:01.501353+0000","last_became_active":"2023-12-04T05:39:59.699479+0000","last_became_peered":"2023-12-04T05:39:59.699479+0000","last_unstale":"2023-12-04T05:40:01.501353+0000","last_undegraded":"2023-12-04T05:40:01.501353+0000","last_fullsized":"2023-12-04T05:40:01.501353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:12:00.122168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683148+0000","last_change":"2023-12-04T05:39:59.728698+0000","last_active":"2023-12-04T05:40:00.683148+0000","last_peered":"2023-12-04T05:40:00.683148+0000","last_clean":"2023-12-04T05:40:00.683148+0000","last_became_active":"2023-12-04T05:39:59.727914+0000","last_became_peered":"2023-12-04T05:39:59.727914+0000","last_unstale":"2023-12-04T05:40:00.683148+0000","last_undegraded":"2023-12-04T05:40:00.683148+0000","last_fullsized":"2023-12-04T05:40:00.683148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:39:17.269322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502651+0000","last_change":"2023-12-04T05:39:59.726320+0000","last_active":"2023-12-04T05:40:01.502651+0000","last_peered":"2023-12-04T05:40:01.502651+0000","last_clean":"2023-12-04T05:40:01.502651+0000","last_became_active":"2023-12-04T05:39:59.722627+0000","last_became_peered":"2023-12-04T05:39:59.722627+0000","last_unstale":"2023-12-04T05:40:01.502651+0000","last_undegraded":"2023-12-04T05:40:01.502651+0000","last_fullsized":"2023-12-04T05:40:01.502651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:40:48.940938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682860+0000","last_change":"2023-12-04T05:39:59.718363+0000","last_active":"2023-12-04T05:40:00.682860+0000","last_peered":"2023-12-04T05:40:00.682860+0000","last_clean":"2023-12-04T05:40:00.682860+0000","last_became_active":"2023-12-04T05:39:59.717475+0000","last_became_peered":"2023-12-04T05:39:59.717475+0000","last_unstale":"2023-12-04T05:40:00.682860+0000","last_undegraded":"2023-12-04T05:40:00.682860+0000","last_fullsized":"2023-12-04T05:40:00.682860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:02:56.684095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501353+0000","last_change":"2023-12-04T05:39:59.724035+0000","last_active":"2023-12-04T05:40:01.501353+0000","last_peered":"2023-12-04T05:40:01.501353+0000","last_clean":"2023-12-04T05:40:01.501353+0000","last_became_active":"2023-12-04T05:39:59.723277+0000","last_became_peered":"2023-12-04T05:39:59.723277+0000","last_unstale":"2023-12-04T05:40:01.501353+0000","last_undegraded":"2023-12-04T05:40:01.501353+0000","last_fullsized":"2023-12-04T05:40:01.501353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:34:54.379064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502157+0000","last_change":"2023-12-04T05:39:59.720028+0000","last_active":"2023-12-04T05:40:01.502157+0000","last_peered":"2023-12-04T05:40:01.502157+0000","last_clean":"2023-12-04T05:40:01.502157+0000","last_became_active":"2023-12-04T05:39:59.719236+0000","last_became_peered":"2023-12-04T05:39:59.719236+0000","last_unstale":"2023-12-04T05:40:01.502157+0000","last_undegraded":"2023-12-04T05:40:01.502157+0000","last_fullsized":"2023-12-04T05:40:01.502157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:23:39.909425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503754+0000","last_change":"2023-12-04T05:39:59.732301+0000","last_active":"2023-12-04T05:40:01.503754+0000","last_peered":"2023-12-04T05:40:01.503754+0000","last_clean":"2023-12-04T05:40:01.503754+0000","last_became_active":"2023-12-04T05:39:59.731561+0000","last_became_peered":"2023-12-04T05:39:59.731561+0000","last_unstale":"2023-12-04T05:40:01.503754+0000","last_undegraded":"2023-12-04T05:40:01.503754+0000","last_fullsized":"2023-12-04T05:40:01.503754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:49:40.179867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501688+0000","last_change":"2023-12-04T05:39:59.721633+0000","last_active":"2023-12-04T05:40:01.501688+0000","last_peered":"2023-12-04T05:40:01.501688+0000","last_clean":"2023-12-04T05:40:01.501688+0000","last_became_active":"2023-12-04T05:39:59.720846+0000","last_became_peered":"2023-12-04T05:39:59.720846+0000","last_unstale":"2023-12-04T05:40:01.501688+0000","last_undegraded":"2023-12-04T05:40:01.501688+0000","last_fullsized":"2023-12-04T05:40:01.501688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:44:34.893307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501835+0000","last_change":"2023-12-04T05:39:59.721173+0000","last_active":"2023-12-04T05:40:01.501835+0000","last_peered":"2023-12-04T05:40:01.501835+0000","last_clean":"2023-12-04T05:40:01.501835+0000","last_became_active":"2023-12-04T05:39:59.720371+0000","last_became_peered":"2023-12-04T05:39:59.720371+0000","last_unstale":"2023-12-04T05:40:01.501835+0000","last_undegraded":"2023-12-04T05:40:01.501835+0000","last_fullsized":"2023-12-04T05:40:01.501835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:12:23.683644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682272+0000","last_change":"2023-12-04T05:39:59.710269+0000","last_active":"2023-12-04T05:40:00.682272+0000","last_peered":"2023-12-04T05:40:00.682272+0000","last_clean":"2023-12-04T05:40:00.682272+0000","last_became_active":"2023-12-04T05:39:59.709565+0000","last_became_peered":"2023-12-04T05:39:59.709565+0000","last_unstale":"2023-12-04T05:40:00.682272+0000","last_undegraded":"2023-12-04T05:40:00.682272+0000","last_fullsized":"2023-12-04T05:40:00.682272+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:11:13.222803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501884+0000","last_change":"2023-12-04T05:39:59.710484+0000","last_active":"2023-12-04T05:40:01.501884+0000","last_peered":"2023-12-04T05:40:01.501884+0000","last_clean":"2023-12-04T05:40:01.501884+0000","last_became_active":"2023-12-04T05:39:59.709623+0000","last_became_peered":"2023-12-04T05:39:59.709623+0000","last_unstale":"2023-12-04T05:40:01.501884+0000","last_undegraded":"2023-12-04T05:40:01.501884+0000","last_fullsized":"2023-12-04T05:40:01.501884+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:40:05.852673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.500880+0000","last_change":"2023-12-04T05:39:59.715685+0000","last_active":"2023-12-04T05:40:01.500880+0000","last_peered":"2023-12-04T05:40:01.500880+0000","last_clean":"2023-12-04T05:40:01.500880+0000","last_became_active":"2023-12-04T05:39:59.714890+0000","last_became_peered":"2023-12-04T05:39:59.714890+0000","last_unstale":"2023-12-04T05:40:01.500880+0000","last_undegraded":"2023-12-04T05:40:01.500880+0000","last_fullsized":"2023-12-04T05:40:01.500880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:59:34.243902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908703+0000","last_change":"2023-12-04T05:39:59.688405+0000","last_active":"2023-12-04T05:40:00.908703+0000","last_peered":"2023-12-04T05:40:00.908703+0000","last_clean":"2023-12-04T05:40:00.908703+0000","last_became_active":"2023-12-04T05:39:59.687603+0000","last_became_peered":"2023-12-04T05:39:59.687603+0000","last_unstale":"2023-12-04T05:40:00.908703+0000","last_undegraded":"2023-12-04T05:40:00.908703+0000","last_fullsized":"2023-12-04T05:40:00.908703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:44:51.269794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681203+0000","last_change":"2023-12-04T05:39:59.718543+0000","last_active":"2023-12-04T05:40:00.681203+0000","last_peered":"2023-12-04T05:40:00.681203+0000","last_clean":"2023-12-04T05:40:00.681203+0000","last_became_active":"2023-12-04T05:39:59.717826+0000","last_became_peered":"2023-12-04T05:39:59.717826+0000","last_unstale":"2023-12-04T05:40:00.681203+0000","last_undegraded":"2023-12-04T05:40:00.681203+0000","last_fullsized":"2023-12-04T05:40:00.681203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:31:03.248114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.684087+0000","last_change":"2023-12-04T05:39:59.724278+0000","last_active":"2023-12-04T05:40:00.684087+0000","last_peered":"2023-12-04T05:40:00.684087+0000","last_clean":"2023-12-04T05:40:00.684087+0000","last_became_active":"2023-12-04T05:39:59.723384+0000","last_became_peered":"2023-12-04T05:39:59.723384+0000","last_unstale":"2023-12-04T05:40:00.684087+0000","last_undegraded":"2023-12-04T05:40:00.684087+0000","last_fullsized":"2023-12-04T05:40:00.684087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:07:20.576260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.679022+0000","last_change":"2023-12-04T05:39:59.711802+0000","last_active":"2023-12-04T05:40:00.679022+0000","last_peered":"2023-12-04T05:40:00.679022+0000","last_clean":"2023-12-04T05:40:00.679022+0000","last_became_active":"2023-12-04T05:39:59.711044+0000","last_became_peered":"2023-12-04T05:39:59.711044+0000","last_unstale":"2023-12-04T05:40:00.679022+0000","last_undegraded":"2023-12-04T05:40:00.679022+0000","last_fullsized":"2023-12-04T05:40:00.679022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:42:28.221010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503300+0000","last_change":"2023-12-04T05:39:59.725226+0000","last_active":"2023-12-04T05:40:01.503300+0000","last_peered":"2023-12-04T05:40:01.503300+0000","last_clean":"2023-12-04T05:40:01.503300+0000","last_became_active":"2023-12-04T05:39:59.724334+0000","last_became_peered":"2023-12-04T05:39:59.724334+0000","last_unstale":"2023-12-04T05:40:01.503300+0000","last_undegraded":"2023-12-04T05:40:01.503300+0000","last_fullsized":"2023-12-04T05:40:01.503300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:48:30.439285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501624+0000","last_change":"2023-12-04T05:39:59.729858+0000","last_active":"2023-12-04T05:40:01.501624+0000","last_peered":"2023-12-04T05:40:01.501624+0000","last_clean":"2023-12-04T05:40:01.501624+0000","last_became_active":"2023-12-04T05:39:59.728989+0000","last_became_peered":"2023-12-04T05:39:59.728989+0000","last_unstale":"2023-12-04T05:40:01.501624+0000","last_undegraded":"2023-12-04T05:40:01.501624+0000","last_fullsized":"2023-12-04T05:40:01.501624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:42:01.099743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501616+0000","last_change":"2023-12-04T05:39:59.724022+0000","last_active":"2023-12-04T05:40:01.501616+0000","last_peered":"2023-12-04T05:40:01.501616+0000","last_clean":"2023-12-04T05:40:01.501616+0000","last_became_active":"2023-12-04T05:39:59.723120+0000","last_became_peered":"2023-12-04T05:39:59.723120+0000","last_unstale":"2023-12-04T05:40:01.501616+0000","last_undegraded":"2023-12-04T05:40:01.501616+0000","last_fullsized":"2023-12-04T05:40:01.501616+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:13:50.048757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683322+0000","last_change":"2023-12-04T05:39:59.714806+0000","last_active":"2023-12-04T05:40:00.683322+0000","last_peered":"2023-12-04T05:40:00.683322+0000","last_clean":"2023-12-04T05:40:00.683322+0000","last_became_active":"2023-12-04T05:39:59.713917+0000","last_became_peered":"2023-12-04T05:39:59.713917+0000","last_unstale":"2023-12-04T05:40:00.683322+0000","last_undegraded":"2023-12-04T05:40:00.683322+0000","last_fullsized":"2023-12-04T05:40:00.683322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:18:03.128562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501966+0000","last_change":"2023-12-04T05:39:59.729098+0000","last_active":"2023-12-04T05:40:01.501966+0000","last_peered":"2023-12-04T05:40:01.501966+0000","last_clean":"2023-12-04T05:40:01.501966+0000","last_became_active":"2023-12-04T05:39:59.728449+0000","last_became_peered":"2023-12-04T05:39:59.728449+0000","last_unstale":"2023-12-04T05:40:01.501966+0000","last_undegraded":"2023-12-04T05:40:01.501966+0000","last_fullsized":"2023-12-04T05:40:01.501966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:25:12.098431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682551+0000","last_change":"2023-12-04T05:39:59.708300+0000","last_active":"2023-12-04T05:40:00.682551+0000","last_peered":"2023-12-04T05:40:00.682551+0000","last_clean":"2023-12-04T05:40:00.682551+0000","last_became_active":"2023-12-04T05:39:59.707530+0000","last_became_peered":"2023-12-04T05:39:59.707530+0000","last_unstale":"2023-12-04T05:40:00.682551+0000","last_undegraded":"2023-12-04T05:40:00.682551+0000","last_fullsized":"2023-12-04T05:40:00.682551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:11:40.053312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502562+0000","last_change":"2023-12-04T05:39:59.729406+0000","last_active":"2023-12-04T05:40:01.502562+0000","last_peered":"2023-12-04T05:40:01.502562+0000","last_clean":"2023-12-04T05:40:01.502562+0000","last_became_active":"2023-12-04T05:39:59.728677+0000","last_became_peered":"2023-12-04T05:39:59.728677+0000","last_unstale":"2023-12-04T05:40:01.502562+0000","last_undegraded":"2023-12-04T05:40:01.502562+0000","last_fullsized":"2023-12-04T05:40:01.502562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:03:26.954387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501922+0000","last_change":"2023-12-04T05:39:59.733628+0000","last_active":"2023-12-04T05:40:01.501922+0000","last_peered":"2023-12-04T05:40:01.501922+0000","last_clean":"2023-12-04T05:40:01.501922+0000","last_became_active":"2023-12-04T05:39:59.726137+0000","last_became_peered":"2023-12-04T05:39:59.726137+0000","last_unstale":"2023-12-04T05:40:01.501922+0000","last_undegraded":"2023-12-04T05:40:01.501922+0000","last_fullsized":"2023-12-04T05:40:01.501922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:49:17.630085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908788+0000","last_change":"2023-12-04T05:39:59.727169+0000","last_active":"2023-12-04T05:40:00.908788+0000","last_peered":"2023-12-04T05:40:00.908788+0000","last_clean":"2023-12-04T05:40:00.908788+0000","last_became_active":"2023-12-04T05:39:59.725378+0000","last_became_peered":"2023-12-04T05:39:59.725378+0000","last_unstale":"2023-12-04T05:40:00.908788+0000","last_undegraded":"2023-12-04T05:40:00.908788+0000","last_fullsized":"2023-12-04T05:40:00.908788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:23:21.007397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502427+0000","last_change":"2023-12-04T05:39:59.729516+0000","last_active":"2023-12-04T05:40:01.502427+0000","last_peered":"2023-12-04T05:40:01.502427+0000","last_clean":"2023-12-04T05:40:01.502427+0000","last_became_active":"2023-12-04T05:39:59.728835+0000","last_became_peered":"2023-12-04T05:39:59.728835+0000","last_unstale":"2023-12-04T05:40:01.502427+0000","last_undegraded":"2023-12-04T05:40:01.502427+0000","last_fullsized":"2023-12-04T05:40:01.502427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:37:37.490362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501933+0000","last_change":"2023-12-04T05:39:59.724592+0000","last_active":"2023-12-04T05:40:01.501933+0000","last_peered":"2023-12-04T05:40:01.501933+0000","last_clean":"2023-12-04T05:40:01.501933+0000","last_became_active":"2023-12-04T05:39:59.723876+0000","last_became_peered":"2023-12-04T05:39:59.723876+0000","last_unstale":"2023-12-04T05:40:01.501933+0000","last_undegraded":"2023-12-04T05:40:01.501933+0000","last_fullsized":"2023-12-04T05:40:01.501933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:09:26.625728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683703+0000","last_change":"2023-12-04T05:39:59.726540+0000","last_active":"2023-12-04T05:40:00.683703+0000","last_peered":"2023-12-04T05:40:00.683703+0000","last_clean":"2023-12-04T05:40:00.683703+0000","last_became_active":"2023-12-04T05:39:59.725781+0000","last_became_peered":"2023-12-04T05:39:59.725781+0000","last_unstale":"2023-12-04T05:40:00.683703+0000","last_undegraded":"2023-12-04T05:40:00.683703+0000","last_fullsized":"2023-12-04T05:40:00.683703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:59:22.013410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.678751+0000","last_change":"2023-12-04T05:39:59.708376+0000","last_active":"2023-12-04T05:40:00.678751+0000","last_peered":"2023-12-04T05:40:00.678751+0000","last_clean":"2023-12-04T05:40:00.678751+0000","last_became_active":"2023-12-04T05:39:59.707604+0000","last_became_peered":"2023-12-04T05:39:59.707604+0000","last_unstale":"2023-12-04T05:40:00.678751+0000","last_undegraded":"2023-12-04T05:40:00.678751+0000","last_fullsized":"2023-12-04T05:40:00.678751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:00:23.385619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501476+0000","last_change":"2023-12-04T05:39:59.720510+0000","last_active":"2023-12-04T05:40:01.501476+0000","last_peered":"2023-12-04T05:40:01.501476+0000","last_clean":"2023-12-04T05:40:01.501476+0000","last_became_active":"2023-12-04T05:39:59.719704+0000","last_became_peered":"2023-12-04T05:39:59.719704+0000","last_unstale":"2023-12-04T05:40:01.501476+0000","last_undegraded":"2023-12-04T05:40:01.501476+0000","last_fullsized":"2023-12-04T05:40:01.501476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:39:15.714817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681790+0000","last_change":"2023-12-04T05:39:59.711839+0000","last_active":"2023-12-04T05:40:00.681790+0000","last_peered":"2023-12-04T05:40:00.681790+0000","last_clean":"2023-12-04T05:40:00.681790+0000","last_became_active":"2023-12-04T05:39:59.711113+0000","last_became_peered":"2023-12-04T05:39:59.711113+0000","last_unstale":"2023-12-04T05:40:00.681790+0000","last_undegraded":"2023-12-04T05:40:00.681790+0000","last_fullsized":"2023-12-04T05:40:00.681790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:21:17.569525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.529158+0000","last_change":"2023-12-04T05:39:58.750554+0000","last_active":"2023-12-04T05:40:01.529158+0000","last_peered":"2023-12-04T05:40:01.529158+0000","last_clean":"2023-12-04T05:40:01.529158+0000","last_became_active":"2023-12-04T05:39:58.750227+0000","last_became_peered":"2023-12-04T05:39:58.750227+0000","last_unstale":"2023-12-04T05:40:01.529158+0000","last_undegraded":"2023-12-04T05:40:01.529158+0000","last_fullsized":"2023-12-04T05:40:01.529158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:13:23.997255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502276+0000","last_change":"2023-12-04T05:39:59.731582+0000","last_active":"2023-12-04T05:40:01.502276+0000","last_peered":"2023-12-04T05:40:01.502276+0000","last_clean":"2023-12-04T05:40:01.502276+0000","last_became_active":"2023-12-04T05:39:59.730822+0000","last_became_peered":"2023-12-04T05:39:59.730822+0000","last_unstale":"2023-12-04T05:40:01.502276+0000","last_undegraded":"2023-12-04T05:40:01.502276+0000","last_fullsized":"2023-12-04T05:40:01.502276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:10:55.996104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.524401+0000","last_change":"2023-12-04T05:39:58.740011+0000","last_active":"2023-12-04T05:40:01.524401+0000","last_peered":"2023-12-04T05:40:01.524401+0000","last_clean":"2023-12-04T05:40:01.524401+0000","last_became_active":"2023-12-04T05:39:58.739395+0000","last_became_peered":"2023-12-04T05:39:58.739395+0000","last_unstale":"2023-12-04T05:40:01.524401+0000","last_undegraded":"2023-12-04T05:40:01.524401+0000","last_fullsized":"2023-12-04T05:40:01.524401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:17:48.107311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502182+0000","last_change":"2023-12-04T05:39:58.730186+0000","last_active":"2023-12-04T05:40:01.502182+0000","last_peered":"2023-12-04T05:40:01.502182+0000","last_clean":"2023-12-04T05:40:01.502182+0000","last_became_active":"2023-12-04T05:39:58.729411+0000","last_became_peered":"2023-12-04T05:39:58.729411+0000","last_unstale":"2023-12-04T05:40:01.502182+0000","last_undegraded":"2023-12-04T05:40:01.502182+0000","last_fullsized":"2023-12-04T05:40:01.502182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:39:05.463010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502530+0000","last_change":"2023-12-04T05:39:59.715596+0000","last_active":"2023-12-04T05:40:01.502530+0000","last_peered":"2023-12-04T05:40:01.502530+0000","last_clean":"2023-12-04T05:40:01.502530+0000","last_became_active":"2023-12-04T05:39:59.714842+0000","last_became_peered":"2023-12-04T05:39:59.714842+0000","last_unstale":"2023-12-04T05:40:01.502530+0000","last_undegraded":"2023-12-04T05:40:01.502530+0000","last_fullsized":"2023-12-04T05:40:01.502530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:51:10.874053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682760+0000","last_change":"2023-12-04T05:39:58.722113+0000","last_active":"2023-12-04T05:40:00.682760+0000","last_peered":"2023-12-04T05:40:00.682760+0000","last_clean":"2023-12-04T05:40:00.682760+0000","last_became_active":"2023-12-04T05:39:58.721346+0000","last_became_peered":"2023-12-04T05:39:58.721346+0000","last_unstale":"2023-12-04T05:40:00.682760+0000","last_undegraded":"2023-12-04T05:40:00.682760+0000","last_fullsized":"2023-12-04T05:40:00.682760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:30:11.578047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908461+0000","last_change":"2023-12-04T05:39:59.708575+0000","last_active":"2023-12-04T05:40:00.908461+0000","last_peered":"2023-12-04T05:40:00.908461+0000","last_clean":"2023-12-04T05:40:00.908461+0000","last_became_active":"2023-12-04T05:39:59.707753+0000","last_became_peered":"2023-12-04T05:39:59.707753+0000","last_unstale":"2023-12-04T05:40:00.908461+0000","last_undegraded":"2023-12-04T05:40:00.908461+0000","last_fullsized":"2023-12-04T05:40:00.908461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:47:58.349368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501888+0000","last_change":"2023-12-04T05:39:58.734739+0000","last_active":"2023-12-04T05:40:01.501888+0000","last_peered":"2023-12-04T05:40:01.501888+0000","last_clean":"2023-12-04T05:40:01.501888+0000","last_became_active":"2023-12-04T05:39:58.733897+0000","last_became_peered":"2023-12-04T05:39:58.733897+0000","last_unstale":"2023-12-04T05:40:01.501888+0000","last_undegraded":"2023-12-04T05:40:01.501888+0000","last_fullsized":"2023-12-04T05:40:01.501888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:52:15.157364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683392+0000","last_change":"2023-12-04T05:39:59.715796+0000","last_active":"2023-12-04T05:40:00.683392+0000","last_peered":"2023-12-04T05:40:00.683392+0000","last_clean":"2023-12-04T05:40:00.683392+0000","last_became_active":"2023-12-04T05:39:59.714951+0000","last_became_peered":"2023-12-04T05:39:59.714951+0000","last_unstale":"2023-12-04T05:40:00.683392+0000","last_undegraded":"2023-12-04T05:40:00.683392+0000","last_fullsized":"2023-12-04T05:40:00.683392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:31:12.475136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683603+0000","last_change":"2023-12-04T05:39:58.725387+0000","last_active":"2023-12-04T05:40:00.683603+0000","last_peered":"2023-12-04T05:40:00.683603+0000","last_clean":"2023-12-04T05:40:00.683603+0000","last_became_active":"2023-12-04T05:39:58.724672+0000","last_became_peered":"2023-12-04T05:39:58.724672+0000","last_unstale":"2023-12-04T05:40:00.683603+0000","last_undegraded":"2023-12-04T05:40:00.683603+0000","last_fullsized":"2023-12-04T05:40:00.683603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:19:43.112172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503316+0000","last_change":"2023-12-04T05:39:59.718789+0000","last_active":"2023-12-04T05:40:01.503316+0000","last_peered":"2023-12-04T05:40:01.503316+0000","last_clean":"2023-12-04T05:40:01.503316+0000","last_became_active":"2023-12-04T05:39:59.717927+0000","last_became_peered":"2023-12-04T05:39:59.717927+0000","last_unstale":"2023-12-04T05:40:01.503316+0000","last_undegraded":"2023-12-04T05:40:01.503316+0000","last_fullsized":"2023-12-04T05:40:01.503316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:43:08.771461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.967861+0000","last_change":"2023-12-04T05:39:58.731710+0000","last_active":"2023-12-04T05:40:04.967861+0000","last_peered":"2023-12-04T05:40:04.967861+0000","last_clean":"2023-12-04T05:40:04.967861+0000","last_became_active":"2023-12-04T05:39:58.731373+0000","last_became_peered":"2023-12-04T05:39:58.731373+0000","last_unstale":"2023-12-04T05:40:04.967861+0000","last_undegraded":"2023-12-04T05:40:04.967861+0000","last_fullsized":"2023-12-04T05:40:04.967861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:03:37.899195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682594+0000","last_change":"2023-12-04T05:39:59.725444+0000","last_active":"2023-12-04T05:40:00.682594+0000","last_peered":"2023-12-04T05:40:00.682594+0000","last_clean":"2023-12-04T05:40:00.682594+0000","last_became_active":"2023-12-04T05:39:59.724620+0000","last_became_peered":"2023-12-04T05:39:59.724620+0000","last_unstale":"2023-12-04T05:40:00.682594+0000","last_undegraded":"2023-12-04T05:40:00.682594+0000","last_fullsized":"2023-12-04T05:40:00.682594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:11:17.092060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.781096+0000","last_change":"2023-12-04T05:39:58.730135+0000","last_active":"2023-12-04T05:40:04.781096+0000","last_peered":"2023-12-04T05:40:04.781096+0000","last_clean":"2023-12-04T05:40:04.781096+0000","last_became_active":"2023-12-04T05:39:58.729295+0000","last_became_peered":"2023-12-04T05:39:58.729295+0000","last_unstale":"2023-12-04T05:40:04.781096+0000","last_undegraded":"2023-12-04T05:40:04.781096+0000","last_fullsized":"2023-12-04T05:40:04.781096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:24:42.912063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502658+0000","last_change":"2023-12-04T05:39:59.731327+0000","last_active":"2023-12-04T05:40:01.502658+0000","last_peered":"2023-12-04T05:40:01.502658+0000","last_clean":"2023-12-04T05:40:01.502658+0000","last_became_active":"2023-12-04T05:39:59.730427+0000","last_became_peered":"2023-12-04T05:39:59.730427+0000","last_unstale":"2023-12-04T05:40:01.502658+0000","last_undegraded":"2023-12-04T05:40:01.502658+0000","last_fullsized":"2023-12-04T05:40:01.502658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:34:45.211951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.767631+0000","last_change":"2023-12-04T05:39:58.711690+0000","last_active":"2023-12-04T05:40:04.767631+0000","last_peered":"2023-12-04T05:40:04.767631+0000","last_clean":"2023-12-04T05:40:04.767631+0000","last_became_active":"2023-12-04T05:39:58.711305+0000","last_became_peered":"2023-12-04T05:39:58.711305+0000","last_unstale":"2023-12-04T05:40:04.767631+0000","last_undegraded":"2023-12-04T05:40:04.767631+0000","last_fullsized":"2023-12-04T05:40:04.767631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:33:02.159265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501048+0000","last_change":"2023-12-04T05:39:59.702650+0000","last_active":"2023-12-04T05:40:01.501048+0000","last_peered":"2023-12-04T05:40:01.501048+0000","last_clean":"2023-12-04T05:40:01.501048+0000","last_became_active":"2023-12-04T05:39:59.701827+0000","last_became_peered":"2023-12-04T05:39:59.701827+0000","last_unstale":"2023-12-04T05:40:01.501048+0000","last_undegraded":"2023-12-04T05:40:01.501048+0000","last_fullsized":"2023-12-04T05:40:01.501048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:46:45.006062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683932+0000","last_change":"2023-12-04T05:39:58.729371+0000","last_active":"2023-12-04T05:40:00.683932+0000","last_peered":"2023-12-04T05:40:00.683932+0000","last_clean":"2023-12-04T05:40:00.683932+0000","last_became_active":"2023-12-04T05:39:58.727781+0000","last_became_peered":"2023-12-04T05:39:58.727781+0000","last_unstale":"2023-12-04T05:40:00.683932+0000","last_undegraded":"2023-12-04T05:40:00.683932+0000","last_fullsized":"2023-12-04T05:40:00.683932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:37:29.707707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502646+0000","last_change":"2023-12-04T05:39:59.742080+0000","last_active":"2023-12-04T05:40:01.502646+0000","last_peered":"2023-12-04T05:40:01.502646+0000","last_clean":"2023-12-04T05:40:01.502646+0000","last_became_active":"2023-12-04T05:39:59.741440+0000","last_became_peered":"2023-12-04T05:39:59.741440+0000","last_unstale":"2023-12-04T05:40:01.502646+0000","last_undegraded":"2023-12-04T05:40:01.502646+0000","last_fullsized":"2023-12-04T05:40:01.502646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:29:21.379583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.500870+0000","last_change":"2023-12-04T05:39:58.737926+0000","last_active":"2023-12-04T05:40:01.500870+0000","last_peered":"2023-12-04T05:40:01.500870+0000","last_clean":"2023-12-04T05:40:01.500870+0000","last_became_active":"2023-12-04T05:39:58.737403+0000","last_became_peered":"2023-12-04T05:39:58.737403+0000","last_unstale":"2023-12-04T05:40:01.500870+0000","last_undegraded":"2023-12-04T05:40:01.500870+0000","last_fullsized":"2023-12-04T05:40:01.500870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:10:59.035282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501005+0000","last_change":"2023-12-04T05:39:59.710310+0000","last_active":"2023-12-04T05:40:01.501005+0000","last_peered":"2023-12-04T05:40:01.501005+0000","last_clean":"2023-12-04T05:40:01.501005+0000","last_became_active":"2023-12-04T05:39:59.709497+0000","last_became_peered":"2023-12-04T05:39:59.709497+0000","last_unstale":"2023-12-04T05:40:01.501005+0000","last_undegraded":"2023-12-04T05:40:01.501005+0000","last_fullsized":"2023-12-04T05:40:01.501005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:43:43.708311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.779639+0000","last_change":"2023-12-04T05:39:58.728046+0000","last_active":"2023-12-04T05:40:04.779639+0000","last_peered":"2023-12-04T05:40:04.779639+0000","last_clean":"2023-12-04T05:40:04.779639+0000","last_became_active":"2023-12-04T05:39:58.726747+0000","last_became_peered":"2023-12-04T05:39:58.726747+0000","last_unstale":"2023-12-04T05:40:04.779639+0000","last_undegraded":"2023-12-04T05:40:04.779639+0000","last_fullsized":"2023-12-04T05:40:04.779639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:45:11.024395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501689+0000","last_change":"2023-12-04T05:39:59.741770+0000","last_active":"2023-12-04T05:40:01.501689+0000","last_peered":"2023-12-04T05:40:01.501689+0000","last_clean":"2023-12-04T05:40:01.501689+0000","last_became_active":"2023-12-04T05:39:59.741091+0000","last_became_peered":"2023-12-04T05:39:59.741091+0000","last_unstale":"2023-12-04T05:40:01.501689+0000","last_undegraded":"2023-12-04T05:40:01.501689+0000","last_fullsized":"2023-12-04T05:40:01.501689+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:59:20.747612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.524182+0000","last_change":"2023-12-04T05:39:58.712970+0000","last_active":"2023-12-04T05:40:01.524182+0000","last_peered":"2023-12-04T05:40:01.524182+0000","last_clean":"2023-12-04T05:40:01.524182+0000","last_became_active":"2023-12-04T05:39:58.711759+0000","last_became_peered":"2023-12-04T05:39:58.711759+0000","last_unstale":"2023-12-04T05:40:01.524182+0000","last_undegraded":"2023-12-04T05:40:01.524182+0000","last_fullsized":"2023-12-04T05:40:01.524182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:02:00.957563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503407+0000","last_change":"2023-12-04T05:39:59.742244+0000","last_active":"2023-12-04T05:40:01.503407+0000","last_peered":"2023-12-04T05:40:01.503407+0000","last_clean":"2023-12-04T05:40:01.503407+0000","last_became_active":"2023-12-04T05:39:59.741379+0000","last_became_peered":"2023-12-04T05:39:59.741379+0000","last_unstale":"2023-12-04T05:40:01.503407+0000","last_undegraded":"2023-12-04T05:40:01.503407+0000","last_fullsized":"2023-12-04T05:40:01.503407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:07:57.685895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.769288+0000","last_change":"2023-12-04T05:39:58.729723+0000","last_active":"2023-12-04T05:40:04.769288+0000","last_peered":"2023-12-04T05:40:04.769288+0000","last_clean":"2023-12-04T05:40:04.769288+0000","last_became_active":"2023-12-04T05:39:58.729410+0000","last_became_peered":"2023-12-04T05:39:58.729410+0000","last_unstale":"2023-12-04T05:40:04.769288+0000","last_undegraded":"2023-12-04T05:40:04.769288+0000","last_fullsized":"2023-12-04T05:40:04.769288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:59:10.799774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503914+0000","last_change":"2023-12-04T05:39:59.741718+0000","last_active":"2023-12-04T05:40:01.503914+0000","last_peered":"2023-12-04T05:40:01.503914+0000","last_clean":"2023-12-04T05:40:01.503914+0000","last_became_active":"2023-12-04T05:39:59.741101+0000","last_became_peered":"2023-12-04T05:39:59.741101+0000","last_unstale":"2023-12-04T05:40:01.503914+0000","last_undegraded":"2023-12-04T05:40:01.503914+0000","last_fullsized":"2023-12-04T05:40:01.503914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:41:03.969161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.521601+0000","last_change":"2023-12-04T05:39:58.737224+0000","last_active":"2023-12-04T05:40:01.521601+0000","last_peered":"2023-12-04T05:40:01.521601+0000","last_clean":"2023-12-04T05:40:01.521601+0000","last_became_active":"2023-12-04T05:39:58.736753+0000","last_became_peered":"2023-12-04T05:39:58.736753+0000","last_unstale":"2023-12-04T05:40:01.521601+0000","last_undegraded":"2023-12-04T05:40:01.521601+0000","last_fullsized":"2023-12-04T05:40:01.521601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:33:34.027796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683405+0000","last_change":"2023-12-04T05:39:59.726776+0000","last_active":"2023-12-04T05:40:00.683405+0000","last_peered":"2023-12-04T05:40:00.683405+0000","last_clean":"2023-12-04T05:40:00.683405+0000","last_became_active":"2023-12-04T05:39:59.725991+0000","last_became_peered":"2023-12-04T05:39:59.725991+0000","last_unstale":"2023-12-04T05:40:00.683405+0000","last_undegraded":"2023-12-04T05:40:00.683405+0000","last_fullsized":"2023-12-04T05:40:00.683405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:36:02.168898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.678822+0000","last_change":"2023-12-04T05:39:58.732942+0000","last_active":"2023-12-04T05:40:00.678822+0000","last_peered":"2023-12-04T05:40:00.678822+0000","last_clean":"2023-12-04T05:40:00.678822+0000","last_became_active":"2023-12-04T05:39:58.732401+0000","last_became_peered":"2023-12-04T05:39:58.732401+0000","last_unstale":"2023-12-04T05:40:00.678822+0000","last_undegraded":"2023-12-04T05:40:00.678822+0000","last_fullsized":"2023-12-04T05:40:00.678822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:23:13.676061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503215+0000","last_change":"2023-12-04T05:39:59.730768+0000","last_active":"2023-12-04T05:40:01.503215+0000","last_peered":"2023-12-04T05:40:01.503215+0000","last_clean":"2023-12-04T05:40:01.503215+0000","last_became_active":"2023-12-04T05:39:59.730207+0000","last_became_peered":"2023-12-04T05:39:59.730207+0000","last_unstale":"2023-12-04T05:40:01.503215+0000","last_undegraded":"2023-12-04T05:40:01.503215+0000","last_fullsized":"2023-12-04T05:40:01.503215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:25:19.473188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T05:39:59.705581+0000","last_change":"2023-12-04T05:39:49.314208+0000","last_active":"2023-12-04T05:39:59.705581+0000","last_peered":"2023-12-04T05:39:59.705581+0000","last_clean":"2023-12-04T05:39:59.705581+0000","last_became_active":"2023-12-04T05:39:49.312262+0000","last_became_peered":"2023-12-04T05:39:49.312262+0000","last_unstale":"2023-12-04T05:39:59.705581+0000","last_undegraded":"2023-12-04T05:39:59.705581+0000","last_fullsized":"2023-12-04T05:39:59.705581+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:48.242612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:48.242612+0000","last_clean_scrub_stamp":"2023-12-04T05:39:48.242612+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:19:26.511345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682745+0000","last_change":"2023-12-04T05:39:58.725538+0000","last_active":"2023-12-04T05:40:00.682745+0000","last_peered":"2023-12-04T05:40:00.682745+0000","last_clean":"2023-12-04T05:40:00.682745+0000","last_became_active":"2023-12-04T05:39:58.724839+0000","last_became_peered":"2023-12-04T05:39:58.724839+0000","last_unstale":"2023-12-04T05:40:00.682745+0000","last_undegraded":"2023-12-04T05:40:00.682745+0000","last_fullsized":"2023-12-04T05:40:00.682745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:34:48.521117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501800+0000","last_change":"2023-12-04T05:39:59.719475+0000","last_active":"2023-12-04T05:40:01.501800+0000","last_peered":"2023-12-04T05:40:01.501800+0000","last_clean":"2023-12-04T05:40:01.501800+0000","last_became_active":"2023-12-04T05:39:59.718727+0000","last_became_peered":"2023-12-04T05:39:59.718727+0000","last_unstale":"2023-12-04T05:40:01.501800+0000","last_undegraded":"2023-12-04T05:40:01.501800+0000","last_fullsized":"2023-12-04T05:40:01.501800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:53:35.787086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501396+0000","last_change":"2023-12-04T05:39:58.751135+0000","last_active":"2023-12-04T05:40:01.501396+0000","last_peered":"2023-12-04T05:40:01.501396+0000","last_clean":"2023-12-04T05:40:01.501396+0000","last_became_active":"2023-12-04T05:39:58.750144+0000","last_became_peered":"2023-12-04T05:39:58.750144+0000","last_unstale":"2023-12-04T05:40:01.501396+0000","last_undegraded":"2023-12-04T05:40:01.501396+0000","last_fullsized":"2023-12-04T05:40:01.501396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:31:32.599659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.500789+0000","last_change":"2023-12-04T05:39:59.729585+0000","last_active":"2023-12-04T05:40:01.500789+0000","last_peered":"2023-12-04T05:40:01.500789+0000","last_clean":"2023-12-04T05:40:01.500789+0000","last_became_active":"2023-12-04T05:39:59.728381+0000","last_became_peered":"2023-12-04T05:39:59.728381+0000","last_unstale":"2023-12-04T05:40:01.500789+0000","last_undegraded":"2023-12-04T05:40:01.500789+0000","last_fullsized":"2023-12-04T05:40:01.500789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:35:11.920755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.678836+0000","last_change":"2023-12-04T05:39:58.731429+0000","last_active":"2023-12-04T05:40:00.678836+0000","last_peered":"2023-12-04T05:40:00.678836+0000","last_clean":"2023-12-04T05:40:00.678836+0000","last_became_active":"2023-12-04T05:39:58.731049+0000","last_became_peered":"2023-12-04T05:39:58.731049+0000","last_unstale":"2023-12-04T05:40:00.678836+0000","last_undegraded":"2023-12-04T05:40:00.678836+0000","last_fullsized":"2023-12-04T05:40:00.678836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:27:16.863460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908531+0000","last_change":"2023-12-04T05:39:59.701324+0000","last_active":"2023-12-04T05:40:00.908531+0000","last_peered":"2023-12-04T05:40:00.908531+0000","last_clean":"2023-12-04T05:40:00.908531+0000","last_became_active":"2023-12-04T05:39:59.700586+0000","last_became_peered":"2023-12-04T05:39:59.700586+0000","last_unstale":"2023-12-04T05:40:00.908531+0000","last_undegraded":"2023-12-04T05:40:00.908531+0000","last_fullsized":"2023-12-04T05:40:00.908531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:37:17.301688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.684152+0000","last_change":"2023-12-04T05:39:58.729635+0000","last_active":"2023-12-04T05:40:00.684152+0000","last_peered":"2023-12-04T05:40:00.684152+0000","last_clean":"2023-12-04T05:40:00.684152+0000","last_became_active":"2023-12-04T05:39:58.728927+0000","last_became_peered":"2023-12-04T05:39:58.728927+0000","last_unstale":"2023-12-04T05:40:00.684152+0000","last_undegraded":"2023-12-04T05:40:00.684152+0000","last_fullsized":"2023-12-04T05:40:00.684152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:58:46.851065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681790+0000","last_change":"2023-12-04T05:39:59.702840+0000","last_active":"2023-12-04T05:40:00.681790+0000","last_peered":"2023-12-04T05:40:00.681790+0000","last_clean":"2023-12-04T05:40:00.681790+0000","last_became_active":"2023-12-04T05:39:59.701652+0000","last_became_peered":"2023-12-04T05:39:59.701652+0000","last_unstale":"2023-12-04T05:40:00.681790+0000","last_undegraded":"2023-12-04T05:40:00.681790+0000","last_fullsized":"2023-12-04T05:40:00.681790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:44:14.733182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908769+0000","last_change":"2023-12-04T05:39:58.721834+0000","last_active":"2023-12-04T05:40:00.908769+0000","last_peered":"2023-12-04T05:40:00.908769+0000","last_clean":"2023-12-04T05:40:00.908769+0000","last_became_active":"2023-12-04T05:39:58.721451+0000","last_became_peered":"2023-12-04T05:39:58.721451+0000","last_unstale":"2023-12-04T05:40:00.908769+0000","last_undegraded":"2023-12-04T05:40:00.908769+0000","last_fullsized":"2023-12-04T05:40:00.908769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:15:52.065853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.684129+0000","last_change":"2023-12-04T05:39:59.731053+0000","last_active":"2023-12-04T05:40:00.684129+0000","last_peered":"2023-12-04T05:40:00.684129+0000","last_clean":"2023-12-04T05:40:00.684129+0000","last_became_active":"2023-12-04T05:39:59.729732+0000","last_became_peered":"2023-12-04T05:39:59.729732+0000","last_unstale":"2023-12-04T05:40:00.684129+0000","last_undegraded":"2023-12-04T05:40:00.684129+0000","last_fullsized":"2023-12-04T05:40:00.684129+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:57:38.581311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503452+0000","last_change":"2023-12-04T05:39:58.751126+0000","last_active":"2023-12-04T05:40:01.503452+0000","last_peered":"2023-12-04T05:40:01.503452+0000","last_clean":"2023-12-04T05:40:01.503452+0000","last_became_active":"2023-12-04T05:39:58.750496+0000","last_became_peered":"2023-12-04T05:39:58.750496+0000","last_unstale":"2023-12-04T05:40:01.503452+0000","last_undegraded":"2023-12-04T05:40:01.503452+0000","last_fullsized":"2023-12-04T05:40:01.503452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:09:22.333993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908579+0000","last_change":"2023-12-04T05:39:59.700032+0000","last_active":"2023-12-04T05:40:00.908579+0000","last_peered":"2023-12-04T05:40:00.908579+0000","last_clean":"2023-12-04T05:40:00.908579+0000","last_became_active":"2023-12-04T05:39:59.699488+0000","last_became_peered":"2023-12-04T05:39:59.699488+0000","last_unstale":"2023-12-04T05:40:00.908579+0000","last_undegraded":"2023-12-04T05:40:00.908579+0000","last_fullsized":"2023-12-04T05:40:00.908579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:59:57.153486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.909014+0000","last_change":"2023-12-04T05:39:58.723126+0000","last_active":"2023-12-04T05:40:00.909014+0000","last_peered":"2023-12-04T05:40:00.909014+0000","last_clean":"2023-12-04T05:40:00.909014+0000","last_became_active":"2023-12-04T05:39:58.722766+0000","last_became_peered":"2023-12-04T05:39:58.722766+0000","last_unstale":"2023-12-04T05:40:00.909014+0000","last_undegraded":"2023-12-04T05:40:00.909014+0000","last_fullsized":"2023-12-04T05:40:00.909014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:28:47.171345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502918+0000","last_change":"2023-12-04T05:39:59.724653+0000","last_active":"2023-12-04T05:40:01.502918+0000","last_peered":"2023-12-04T05:40:01.502918+0000","last_clean":"2023-12-04T05:40:01.502918+0000","last_became_active":"2023-12-04T05:39:59.723945+0000","last_became_peered":"2023-12-04T05:39:59.723945+0000","last_unstale":"2023-12-04T05:40:01.502918+0000","last_undegraded":"2023-12-04T05:40:01.502918+0000","last_fullsized":"2023-12-04T05:40:01.502918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:13:56.812140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.772912+0000","last_change":"2023-12-04T05:39:58.746966+0000","last_active":"2023-12-04T05:40:04.772912+0000","last_peered":"2023-12-04T05:40:04.772912+0000","last_clean":"2023-12-04T05:40:04.772912+0000","last_became_active":"2023-12-04T05:39:58.746261+0000","last_became_peered":"2023-12-04T05:39:58.746261+0000","last_unstale":"2023-12-04T05:40:04.772912+0000","last_undegraded":"2023-12-04T05:40:04.772912+0000","last_fullsized":"2023-12-04T05:40:04.772912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:57:46.899291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503381+0000","last_change":"2023-12-04T05:39:59.713259+0000","last_active":"2023-12-04T05:40:01.503381+0000","last_peered":"2023-12-04T05:40:01.503381+0000","last_clean":"2023-12-04T05:40:01.503381+0000","last_became_active":"2023-12-04T05:39:59.712523+0000","last_became_peered":"2023-12-04T05:39:59.712523+0000","last_unstale":"2023-12-04T05:40:01.503381+0000","last_undegraded":"2023-12-04T05:40:01.503381+0000","last_fullsized":"2023-12-04T05:40:01.503381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:41:10.001755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501174+0000","last_change":"2023-12-04T05:39:58.750282+0000","last_active":"2023-12-04T05:40:01.501174+0000","last_peered":"2023-12-04T05:40:01.501174+0000","last_clean":"2023-12-04T05:40:01.501174+0000","last_became_active":"2023-12-04T05:39:58.749651+0000","last_became_peered":"2023-12-04T05:39:58.749651+0000","last_unstale":"2023-12-04T05:40:01.501174+0000","last_undegraded":"2023-12-04T05:40:01.501174+0000","last_fullsized":"2023-12-04T05:40:01.501174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:40:55.086649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681687+0000","last_change":"2023-12-04T05:39:59.704667+0000","last_active":"2023-12-04T05:40:00.681687+0000","last_peered":"2023-12-04T05:40:00.681687+0000","last_clean":"2023-12-04T05:40:00.681687+0000","last_became_active":"2023-12-04T05:39:59.703943+0000","last_became_peered":"2023-12-04T05:39:59.703943+0000","last_unstale":"2023-12-04T05:40:00.681687+0000","last_undegraded":"2023-12-04T05:40:00.681687+0000","last_fullsized":"2023-12-04T05:40:00.681687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:31:53.890532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908698+0000","last_change":"2023-12-04T05:39:58.745277+0000","last_active":"2023-12-04T05:40:00.908698+0000","last_peered":"2023-12-04T05:40:00.908698+0000","last_clean":"2023-12-04T05:40:00.908698+0000","last_became_active":"2023-12-04T05:39:58.744842+0000","last_became_peered":"2023-12-04T05:39:58.744842+0000","last_unstale":"2023-12-04T05:40:00.908698+0000","last_undegraded":"2023-12-04T05:40:00.908698+0000","last_fullsized":"2023-12-04T05:40:00.908698+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:50:21.084855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502964+0000","last_change":"2023-12-04T05:39:59.714217+0000","last_active":"2023-12-04T05:40:01.502964+0000","last_peered":"2023-12-04T05:40:01.502964+0000","last_clean":"2023-12-04T05:40:01.502964+0000","last_became_active":"2023-12-04T05:39:59.713438+0000","last_became_peered":"2023-12-04T05:39:59.713438+0000","last_unstale":"2023-12-04T05:40:01.502964+0000","last_undegraded":"2023-12-04T05:40:01.502964+0000","last_fullsized":"2023-12-04T05:40:01.502964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:37:30.790544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908612+0000","last_change":"2023-12-04T05:39:58.722048+0000","last_active":"2023-12-04T05:40:00.908612+0000","last_peered":"2023-12-04T05:40:00.908612+0000","last_clean":"2023-12-04T05:40:00.908612+0000","last_became_active":"2023-12-04T05:39:58.721782+0000","last_became_peered":"2023-12-04T05:39:58.721782+0000","last_unstale":"2023-12-04T05:40:00.908612+0000","last_undegraded":"2023-12-04T05:40:00.908612+0000","last_fullsized":"2023-12-04T05:40:00.908612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:38:16.023373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502222+0000","last_change":"2023-12-04T05:39:59.715313+0000","last_active":"2023-12-04T05:40:01.502222+0000","last_peered":"2023-12-04T05:40:01.502222+0000","last_clean":"2023-12-04T05:40:01.502222+0000","last_became_active":"2023-12-04T05:39:59.714430+0000","last_became_peered":"2023-12-04T05:39:59.714430+0000","last_unstale":"2023-12-04T05:40:01.502222+0000","last_undegraded":"2023-12-04T05:40:01.502222+0000","last_fullsized":"2023-12-04T05:40:01.502222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:10:01.533077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.769466+0000","last_change":"2023-12-04T05:39:58.750921+0000","last_active":"2023-12-04T05:40:04.769466+0000","last_peered":"2023-12-04T05:40:04.769466+0000","last_clean":"2023-12-04T05:40:04.769466+0000","last_became_active":"2023-12-04T05:39:58.750288+0000","last_became_peered":"2023-12-04T05:39:58.750288+0000","last_unstale":"2023-12-04T05:40:04.769466+0000","last_undegraded":"2023-12-04T05:40:04.769466+0000","last_fullsized":"2023-12-04T05:40:04.769466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:04:26.373696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501279+0000","last_change":"2023-12-04T05:39:59.729260+0000","last_active":"2023-12-04T05:40:01.501279+0000","last_peered":"2023-12-04T05:40:01.501279+0000","last_clean":"2023-12-04T05:40:01.501279+0000","last_became_active":"2023-12-04T05:39:59.728639+0000","last_became_peered":"2023-12-04T05:39:59.728639+0000","last_unstale":"2023-12-04T05:40:01.501279+0000","last_undegraded":"2023-12-04T05:40:01.501279+0000","last_fullsized":"2023-12-04T05:40:01.501279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:21:38.039584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683814+0000","last_change":"2023-12-04T05:39:59.731652+0000","last_active":"2023-12-04T05:40:00.683814+0000","last_peered":"2023-12-04T05:40:00.683814+0000","last_clean":"2023-12-04T05:40:00.683814+0000","last_became_active":"2023-12-04T05:39:59.730714+0000","last_became_peered":"2023-12-04T05:39:59.730714+0000","last_unstale":"2023-12-04T05:40:00.683814+0000","last_undegraded":"2023-12-04T05:40:00.683814+0000","last_fullsized":"2023-12-04T05:40:00.683814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:02:27.013221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.769202+0000","last_change":"2023-12-04T05:39:58.751530+0000","last_active":"2023-12-04T05:40:04.769202+0000","last_peered":"2023-12-04T05:40:04.769202+0000","last_clean":"2023-12-04T05:40:04.769202+0000","last_became_active":"2023-12-04T05:39:58.750857+0000","last_became_peered":"2023-12-04T05:39:58.750857+0000","last_unstale":"2023-12-04T05:40:04.769202+0000","last_undegraded":"2023-12-04T05:40:04.769202+0000","last_fullsized":"2023-12-04T05:40:04.769202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:43:36.369813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683195+0000","last_change":"2023-12-04T05:39:59.730535+0000","last_active":"2023-12-04T05:40:00.683195+0000","last_peered":"2023-12-04T05:40:00.683195+0000","last_clean":"2023-12-04T05:40:00.683195+0000","last_became_active":"2023-12-04T05:39:59.729717+0000","last_became_peered":"2023-12-04T05:39:59.729717+0000","last_unstale":"2023-12-04T05:40:00.683195+0000","last_undegraded":"2023-12-04T05:40:00.683195+0000","last_fullsized":"2023-12-04T05:40:00.683195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:31:46.120414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.770150+0000","last_change":"2023-12-04T05:39:58.752193+0000","last_active":"2023-12-04T05:40:04.770150+0000","last_peered":"2023-12-04T05:40:04.770150+0000","last_clean":"2023-12-04T05:40:04.770150+0000","last_became_active":"2023-12-04T05:39:58.751625+0000","last_became_peered":"2023-12-04T05:39:58.751625+0000","last_unstale":"2023-12-04T05:40:04.770150+0000","last_undegraded":"2023-12-04T05:40:04.770150+0000","last_fullsized":"2023-12-04T05:40:04.770150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:05:05.734494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501167+0000","last_change":"2023-12-04T05:39:59.720197+0000","last_active":"2023-12-04T05:40:01.501167+0000","last_peered":"2023-12-04T05:40:01.501167+0000","last_clean":"2023-12-04T05:40:01.501167+0000","last_became_active":"2023-12-04T05:39:59.718736+0000","last_became_peered":"2023-12-04T05:39:59.718736+0000","last_unstale":"2023-12-04T05:40:01.501167+0000","last_undegraded":"2023-12-04T05:40:01.501167+0000","last_fullsized":"2023-12-04T05:40:01.501167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:41:13.184198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.529254+0000","last_change":"2023-12-04T05:39:58.726906+0000","last_active":"2023-12-04T05:40:01.529254+0000","last_peered":"2023-12-04T05:40:01.529254+0000","last_clean":"2023-12-04T05:40:01.529254+0000","last_became_active":"2023-12-04T05:39:58.726415+0000","last_became_peered":"2023-12-04T05:39:58.726415+0000","last_unstale":"2023-12-04T05:40:01.529254+0000","last_undegraded":"2023-12-04T05:40:01.529254+0000","last_fullsized":"2023-12-04T05:40:01.529254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:58:53.715291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503085+0000","last_change":"2023-12-04T05:39:59.711861+0000","last_active":"2023-12-04T05:40:01.503085+0000","last_peered":"2023-12-04T05:40:01.503085+0000","last_clean":"2023-12-04T05:40:01.503085+0000","last_became_active":"2023-12-04T05:39:59.710933+0000","last_became_peered":"2023-12-04T05:39:59.710933+0000","last_unstale":"2023-12-04T05:40:01.503085+0000","last_undegraded":"2023-12-04T05:40:01.503085+0000","last_fullsized":"2023-12-04T05:40:01.503085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:40:46.361768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.771740+0000","last_change":"2023-12-04T05:39:58.740704+0000","last_active":"2023-12-04T05:40:04.771740+0000","last_peered":"2023-12-04T05:40:04.771740+0000","last_clean":"2023-12-04T05:40:04.771740+0000","last_became_active":"2023-12-04T05:39:58.740440+0000","last_became_peered":"2023-12-04T05:39:58.740440+0000","last_unstale":"2023-12-04T05:40:04.771740+0000","last_undegraded":"2023-12-04T05:40:04.771740+0000","last_fullsized":"2023-12-04T05:40:04.771740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:20:18.197917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":815104,"data_stored":666531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.111}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0169999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":221184,"data_stored":75671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1859999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.119}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2789999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.667}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":229376,"data_stored":75707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7629999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.222}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1279999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0880000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":75741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0649999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1200000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":823296,"data_stored":666587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1619999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.22}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-04T05:40:08.422 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-04T05:40:08.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.571+0000 7efcfd050640 1 Processor -- start 2023-12-04T05:40:08.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.571+0000 7efcfd050640 1 -- start start 2023-12-04T05:40:08.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcfd050640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7efcf8071c00 0x7efcf80c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:08.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcfd050640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efcf8070380 0x7efcf80707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:08.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcfd050640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf80700c0 con 0x7efcf8151180 2023-12-04T05:40:08.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcfd050640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcf8156df0 con 0x7efcf8071c00 2023-12-04T05:40:08.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcfd050640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf81570d0 con 0x7efcf8070380 2023-12-04T05:40:08.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf6d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7efcf8151180 0x7efcf80714f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43562/0 (socket says 172.21.15.107:43562) 2023-12-04T05:40:08.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf6d9d640 1 -- 172.21.15.107:0/2468102991 learned_addr learned my addr 172.21.15.107:0/2468102991 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:08.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf759e640 1 --2- 172.21.15.107:0/2468102991 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efcf8070380 0x7efcf80707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:08.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf659c640 1 --2- 172.21.15.107:0/2468102991 >> v2:172.21.15.195:3300/0 conn(0x7efcf8071c00 0x7efcf80c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:08.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf5d9b640 1 -- 172.21.15.107:0/2468102991 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627470848 0 0) 0x7efcf80700c0 con 0x7efcf8151180 2023-12-04T05:40:08.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf5d9b640 1 -- 172.21.15.107:0/2468102991 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efce0003530 con 0x7efcf8151180 2023-12-04T05:40:08.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf5d9b640 1 -- 172.21.15.107:0/2468102991 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2489794406 0 0) 0x7efce0003530 con 0x7efcf8151180 2023-12-04T05:40:08.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf5d9b640 1 -- 172.21.15.107:0/2468102991 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efcf8070380 msgr2=0x7efcf80707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:08.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf5d9b640 1 --2- 172.21.15.107:0/2468102991 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efcf8070380 0x7efcf80707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf5d9b640 1 -- 172.21.15.107:0/2468102991 >> v2:172.21.15.195:3300/0 conn(0x7efcf8071c00 msgr2=0x7efcf80c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:08.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf5d9b640 1 --2- 172.21.15.107:0/2468102991 >> v2:172.21.15.195:3300/0 conn(0x7efcf8071c00 0x7efcf80c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf5d9b640 1 -- 172.21.15.107:0/2468102991 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf81573b0 con 0x7efcf8151180 2023-12-04T05:40:08.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf5d9b640 1 -- 172.21.15.107:0/2468102991 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7efce80016a0 con 0x7efcf8151180 2023-12-04T05:40:08.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf5d9b640 1 -- 172.21.15.107:0/2468102991 >> v1:172.21.15.107:6789/0 conn(0x7efcf8151180 legacy=0x7efcf80714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf5d9b640 1 --2- 172.21.15.107:0/2468102991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce0003980 0x7efce0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:08.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf5d9b640 1 --2- 172.21.15.107:0/2468102991 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efcf8070380 0x7efce00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:08.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf5d9b640 1 --2- 172.21.15.107:0/2468102991 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efce0004d10 0x7efce0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:08.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf5d9b640 1 -- 172.21.15.107:0/2468102991 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf8156df0 con 0x7efce0003980 2023-12-04T05:40:08.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf5d9b640 1 -- 172.21.15.107:0/2468102991 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf81570d0 con 0x7efce0004d10 2023-12-04T05:40:08.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf5d9b640 1 -- 172.21.15.107:0/2468102991 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce000b990 con 0x7efcf8070380 2023-12-04T05:40:08.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf6d9d640 1 --2- 172.21.15.107:0/2468102991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce0003980 0x7efce0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:08.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf659c640 1 --2- 172.21.15.107:0/2468102991 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efcf8070380 0x7efce00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:08.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf6d9d640 1 -- 172.21.15.107:0/2468102991 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efcf8070380 msgr2=0x7efce00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:08.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf6d9d640 1 --2- 172.21.15.107:0/2468102991 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efcf8070380 0x7efce00047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf6d9d640 1 -- 172.21.15.107:0/2468102991 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efce0004d10 msgr2=0x7efce0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:08.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf6d9d640 1 --2- 172.21.15.107:0/2468102991 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efce0004d10 0x7efce0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.575+0000 7efcf6d9d640 1 -- 172.21.15.107:0/2468102991 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce0003530 con 0x7efce0003980 2023-12-04T05:40:08.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcf6d9d640 1 --2- 172.21.15.107:0/2468102991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce0003980 0x7efce0003da0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efce800c210 tx=0x7efce801f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8434785e10c53853 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:08.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcf5d9b640 1 -- 172.21.15.107:0/2468102991 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce8005ae0 con 0x7efce0003980 2023-12-04T05:40:08.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcf5d9b640 1 -- 172.21.15.107:0/2468102991 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce80ab070 con 0x7efce0003980 2023-12-04T05:40:08.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcf5d9b640 1 -- 172.21.15.107:0/2468102991 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce8005ea0 con 0x7efce0003980 2023-12-04T05:40:08.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcfd050640 1 -- 172.21.15.107:0/2468102991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce0003980 msgr2=0x7efce0003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcfd050640 1 --2- 172.21.15.107:0/2468102991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce0003980 0x7efce0003da0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efce800c210 tx=0x7efce801f390 comp rx=0 tx=0).stop 2023-12-04T05:40:08.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcf759e640 1 -- 172.21.15.107:0/2468102991 reap_dead start 2023-12-04T05:40:08.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcfd050640 1 -- 172.21.15.107:0/2468102991 shutdown_connections 2023-12-04T05:40:08.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcfd050640 1 -- 172.21.15.107:0/2468102991 >> 172.21.15.107:0/2468102991 conn(0x7efcf80c7ff0 msgr2=0x7efcf80c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:08.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcfd050640 1 -- 172.21.15.107:0/2468102991 shutdown_connections 2023-12-04T05:40:08.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcfd050640 1 -- 172.21.15.107:0/2468102991 wait complete. 2023-12-04T05:40:08.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcfd050640 1 Processor -- start 2023-12-04T05:40:08.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcfd050640 1 -- start start 2023-12-04T05:40:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcfd050640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce0003980 0x7efcf81554d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcfd050640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efcf8162600 0x7efcf80c1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcfd050640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efcf81f3f60 0x7efcf81f43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcfd050640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf8156580 con 0x7efce0003980 2023-12-04T05:40:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcfd050640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf81f7f30 con 0x7efcf8162600 2023-12-04T05:40:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcfd050640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf81f83c0 con 0x7efcf81f3f60 2023-12-04T05:40:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcf6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce0003980 0x7efcf81554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcf6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce0003980 0x7efcf81554d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40926/0 (socket says 172.21.15.107:40926) 2023-12-04T05:40:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcf6d9d640 1 -- 172.21.15.107:0/2606765013 learned_addr learned my addr 172.21.15.107:0/2606765013 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcf659c640 1 --2- 172.21.15.107:0/2606765013 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efcf8162600 0x7efcf80c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcf759e640 1 --2- 172.21.15.107:0/2606765013 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efcf81f3f60 0x7efcf81f43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcf6d9d640 1 -- 172.21.15.107:0/2606765013 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efcf81f3f60 msgr2=0x7efcf81f43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcf6d9d640 1 --2- 172.21.15.107:0/2606765013 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efcf81f3f60 0x7efcf81f43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcf6d9d640 1 -- 172.21.15.107:0/2606765013 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efcf8162600 msgr2=0x7efcf80c1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcf6d9d640 1 --2- 172.21.15.107:0/2606765013 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efcf8162600 0x7efcf80c1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcf6d9d640 1 -- 172.21.15.107:0/2606765013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf80cb190 con 0x7efce0003980 2023-12-04T05:40:08.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcf6d9d640 1 --2- 172.21.15.107:0/2606765013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce0003980 0x7efcf81554d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7efce8026000 tx=0x7efce0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:08.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcdf7fe640 1 -- 172.21.15.107:0/2606765013 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce80b7070 con 0x7efce0003980 2023-12-04T05:40:08.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcdf7fe640 1 -- 172.21.15.107:0/2606765013 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce8001c80 con 0x7efce0003980 2023-12-04T05:40:08.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.579+0000 7efcdf7fe640 1 -- 172.21.15.107:0/2606765013 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce80016c0 con 0x7efce0003980 2023-12-04T05:40:08.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.583+0000 7efcfd050640 1 -- 172.21.15.107:0/2606765013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf80caeb0 con 0x7efce0003980 2023-12-04T05:40:08.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.583+0000 7efcfd050640 1 -- 172.21.15.107:0/2606765013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf81f78e0 con 0x7efce0003980 2023-12-04T05:40:08.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.583+0000 7efcfd050640 1 -- 172.21.15.107:0/2606765013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf8067870 con 0x7efce0003980 2023-12-04T05:40:08.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.583+0000 7efcdf7fe640 1 -- 172.21.15.107:0/2606765013 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7efce80ad020 con 0x7efce0003980 2023-12-04T05:40:08.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.583+0000 7efcdf7fe640 1 --2- 172.21.15.107:0/2606765013 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efce40668d0 0x7efce4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:08.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.583+0000 7efcdf7fe640 1 -- 172.21.15.107:0/2606765013 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efce8120fe0 con 0x7efce0003980 2023-12-04T05:40:08.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.583+0000 7efcf659c640 1 --2- 172.21.15.107:0/2606765013 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efce40668d0 0x7efce4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:08.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.583+0000 7efcf659c640 1 --2- 172.21.15.107:0/2606765013 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efce40668d0 0x7efce4068d90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efcf806b700 tx=0x7efcec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:08.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.587+0000 7efcdf7fe640 1 -- 172.21.15.107:0/2606765013 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7efce80ee2b0 con 0x7efce0003980 2023-12-04T05:40:08.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.727+0000 7efcfd050640 1 -- 172.21.15.107:0/2606765013 --> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efcf8073130 con 0x7efce40668d0 2023-12-04T05:40:08.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.731+0000 7efcdf7fe640 1 -- 172.21.15.107:0/2606765013 <== mgr.4104 v2:172.21.15.107:6832/2693048026 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264503 (secure 0 0 0) 0x7efcec0096a0 con 0x7efce40668d0 2023-12-04T05:40:08.737 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-12-04T05:40:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.735+0000 7efcfd050640 1 -- 172.21.15.107:0/2606765013 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efce40668d0 msgr2=0x7efce4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.735+0000 7efcfd050640 1 --2- 172.21.15.107:0/2606765013 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efce40668d0 0x7efce4068d90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efcf806b700 tx=0x7efcec008040 comp rx=0 tx=0).stop 2023-12-04T05:40:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.735+0000 7efcfd050640 1 -- 172.21.15.107:0/2606765013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce0003980 msgr2=0x7efcf81554d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.735+0000 7efcfd050640 1 --2- 172.21.15.107:0/2606765013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce0003980 0x7efcf81554d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7efce8026000 tx=0x7efce0005800 comp rx=0 tx=0).stop 2023-12-04T05:40:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.735+0000 7efcfd050640 1 -- 172.21.15.107:0/2606765013 shutdown_connections 2023-12-04T05:40:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.735+0000 7efcfd050640 1 --2- 172.21.15.107:0/2606765013 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7efce40668d0 0x7efce4068d90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.735+0000 7efcfd050640 1 --2- 172.21.15.107:0/2606765013 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7efcf81f3f60 0x7efcf81f43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.735+0000 7efcfd050640 1 --2- 172.21.15.107:0/2606765013 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efcf8162600 0x7efcf80c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.735+0000 7efcfd050640 1 --2- 172.21.15.107:0/2606765013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efce0003980 0x7efcf81554d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.735+0000 7efcfd050640 1 -- 172.21.15.107:0/2606765013 >> 172.21.15.107:0/2606765013 conn(0x7efcf80c7ff0 msgr2=0x7efcf80c59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:08.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.735+0000 7efcfd050640 1 -- 172.21.15.107:0/2606765013 shutdown_connections 2023-12-04T05:40:08.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.735+0000 7efcfd050640 1 -- 172.21.15.107:0/2606765013 wait complete. 2023-12-04T05:40:08.755 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-12-04T05:40:07.493008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235688,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739032,"statfs":{"total":773094113280,"available":772852768768,"internally_reserved":0,"allocated":2990080,"data_stored":1790204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002604"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503577+0000","last_change":"2023-12-04T05:39:59.736570+0000","last_active":"2023-12-04T05:40:01.503577+0000","last_peered":"2023-12-04T05:40:01.503577+0000","last_clean":"2023-12-04T05:40:01.503577+0000","last_became_active":"2023-12-04T05:39:59.736000+0000","last_became_peered":"2023-12-04T05:39:59.736000+0000","last_unstale":"2023-12-04T05:40:01.503577+0000","last_undegraded":"2023-12-04T05:40:01.503577+0000","last_fullsized":"2023-12-04T05:40:01.503577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:34:15.458889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908781+0000","last_change":"2023-12-04T05:39:59.703955+0000","last_active":"2023-12-04T05:40:00.908781+0000","last_peered":"2023-12-04T05:40:00.908781+0000","last_clean":"2023-12-04T05:40:00.908781+0000","last_became_active":"2023-12-04T05:39:59.703032+0000","last_became_peered":"2023-12-04T05:39:59.703032+0000","last_unstale":"2023-12-04T05:40:00.908781+0000","last_undegraded":"2023-12-04T05:40:00.908781+0000","last_fullsized":"2023-12-04T05:40:00.908781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:22:09.010348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501353+0000","last_change":"2023-12-04T05:39:59.705340+0000","last_active":"2023-12-04T05:40:01.501353+0000","last_peered":"2023-12-04T05:40:01.501353+0000","last_clean":"2023-12-04T05:40:01.501353+0000","last_became_active":"2023-12-04T05:39:59.699479+0000","last_became_peered":"2023-12-04T05:39:59.699479+0000","last_unstale":"2023-12-04T05:40:01.501353+0000","last_undegraded":"2023-12-04T05:40:01.501353+0000","last_fullsized":"2023-12-04T05:40:01.501353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:12:00.122168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683148+0000","last_change":"2023-12-04T05:39:59.728698+0000","last_active":"2023-12-04T05:40:00.683148+0000","last_peered":"2023-12-04T05:40:00.683148+0000","last_clean":"2023-12-04T05:40:00.683148+0000","last_became_active":"2023-12-04T05:39:59.727914+0000","last_became_peered":"2023-12-04T05:39:59.727914+0000","last_unstale":"2023-12-04T05:40:00.683148+0000","last_undegraded":"2023-12-04T05:40:00.683148+0000","last_fullsized":"2023-12-04T05:40:00.683148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:39:17.269322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502651+0000","last_change":"2023-12-04T05:39:59.726320+0000","last_active":"2023-12-04T05:40:01.502651+0000","last_peered":"2023-12-04T05:40:01.502651+0000","last_clean":"2023-12-04T05:40:01.502651+0000","last_became_active":"2023-12-04T05:39:59.722627+0000","last_became_peered":"2023-12-04T05:39:59.722627+0000","last_unstale":"2023-12-04T05:40:01.502651+0000","last_undegraded":"2023-12-04T05:40:01.502651+0000","last_fullsized":"2023-12-04T05:40:01.502651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:40:48.940938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682860+0000","last_change":"2023-12-04T05:39:59.718363+0000","last_active":"2023-12-04T05:40:00.682860+0000","last_peered":"2023-12-04T05:40:00.682860+0000","last_clean":"2023-12-04T05:40:00.682860+0000","last_became_active":"2023-12-04T05:39:59.717475+0000","last_became_peered":"2023-12-04T05:39:59.717475+0000","last_unstale":"2023-12-04T05:40:00.682860+0000","last_undegraded":"2023-12-04T05:40:00.682860+0000","last_fullsized":"2023-12-04T05:40:00.682860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:02:56.684095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501353+0000","last_change":"2023-12-04T05:39:59.724035+0000","last_active":"2023-12-04T05:40:01.501353+0000","last_peered":"2023-12-04T05:40:01.501353+0000","last_clean":"2023-12-04T05:40:01.501353+0000","last_became_active":"2023-12-04T05:39:59.723277+0000","last_became_peered":"2023-12-04T05:39:59.723277+0000","last_unstale":"2023-12-04T05:40:01.501353+0000","last_undegraded":"2023-12-04T05:40:01.501353+0000","last_fullsized":"2023-12-04T05:40:01.501353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:34:54.379064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502157+0000","last_change":"2023-12-04T05:39:59.720028+0000","last_active":"2023-12-04T05:40:01.502157+0000","last_peered":"2023-12-04T05:40:01.502157+0000","last_clean":"2023-12-04T05:40:01.502157+0000","last_became_active":"2023-12-04T05:39:59.719236+0000","last_became_peered":"2023-12-04T05:39:59.719236+0000","last_unstale":"2023-12-04T05:40:01.502157+0000","last_undegraded":"2023-12-04T05:40:01.502157+0000","last_fullsized":"2023-12-04T05:40:01.502157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:23:39.909425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503754+0000","last_change":"2023-12-04T05:39:59.732301+0000","last_active":"2023-12-04T05:40:01.503754+0000","last_peered":"2023-12-04T05:40:01.503754+0000","last_clean":"2023-12-04T05:40:01.503754+0000","last_became_active":"2023-12-04T05:39:59.731561+0000","last_became_peered":"2023-12-04T05:39:59.731561+0000","last_unstale":"2023-12-04T05:40:01.503754+0000","last_undegraded":"2023-12-04T05:40:01.503754+0000","last_fullsized":"2023-12-04T05:40:01.503754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:49:40.179867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501688+0000","last_change":"2023-12-04T05:39:59.721633+0000","last_active":"2023-12-04T05:40:01.501688+0000","last_peered":"2023-12-04T05:40:01.501688+0000","last_clean":"2023-12-04T05:40:01.501688+0000","last_became_active":"2023-12-04T05:39:59.720846+0000","last_became_peered":"2023-12-04T05:39:59.720846+0000","last_unstale":"2023-12-04T05:40:01.501688+0000","last_undegraded":"2023-12-04T05:40:01.501688+0000","last_fullsized":"2023-12-04T05:40:01.501688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:44:34.893307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501835+0000","last_change":"2023-12-04T05:39:59.721173+0000","last_active":"2023-12-04T05:40:01.501835+0000","last_peered":"2023-12-04T05:40:01.501835+0000","last_clean":"2023-12-04T05:40:01.501835+0000","last_became_active":"2023-12-04T05:39:59.720371+0000","last_became_peered":"2023-12-04T05:39:59.720371+0000","last_unstale":"2023-12-04T05:40:01.501835+0000","last_undegraded":"2023-12-04T05:40:01.501835+0000","last_fullsized":"2023-12-04T05:40:01.501835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:12:23.683644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682272+0000","last_change":"2023-12-04T05:39:59.710269+0000","last_active":"2023-12-04T05:40:00.682272+0000","last_peered":"2023-12-04T05:40:00.682272+0000","last_clean":"2023-12-04T05:40:00.682272+0000","last_became_active":"2023-12-04T05:39:59.709565+0000","last_became_peered":"2023-12-04T05:39:59.709565+0000","last_unstale":"2023-12-04T05:40:00.682272+0000","last_undegraded":"2023-12-04T05:40:00.682272+0000","last_fullsized":"2023-12-04T05:40:00.682272+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:11:13.222803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501884+0000","last_change":"2023-12-04T05:39:59.710484+0000","last_active":"2023-12-04T05:40:01.501884+0000","last_peered":"2023-12-04T05:40:01.501884+0000","last_clean":"2023-12-04T05:40:01.501884+0000","last_became_active":"2023-12-04T05:39:59.709623+0000","last_became_peered":"2023-12-04T05:39:59.709623+0000","last_unstale":"2023-12-04T05:40:01.501884+0000","last_undegraded":"2023-12-04T05:40:01.501884+0000","last_fullsized":"2023-12-04T05:40:01.501884+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:40:05.852673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.500880+0000","last_change":"2023-12-04T05:39:59.715685+0000","last_active":"2023-12-04T05:40:01.500880+0000","last_peered":"2023-12-04T05:40:01.500880+0000","last_clean":"2023-12-04T05:40:01.500880+0000","last_became_active":"2023-12-04T05:39:59.714890+0000","last_became_peered":"2023-12-04T05:39:59.714890+0000","last_unstale":"2023-12-04T05:40:01.500880+0000","last_undegraded":"2023-12-04T05:40:01.500880+0000","last_fullsized":"2023-12-04T05:40:01.500880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:59:34.243902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908703+0000","last_change":"2023-12-04T05:39:59.688405+0000","last_active":"2023-12-04T05:40:00.908703+0000","last_peered":"2023-12-04T05:40:00.908703+0000","last_clean":"2023-12-04T05:40:00.908703+0000","last_became_active":"2023-12-04T05:39:59.687603+0000","last_became_peered":"2023-12-04T05:39:59.687603+0000","last_unstale":"2023-12-04T05:40:00.908703+0000","last_undegraded":"2023-12-04T05:40:00.908703+0000","last_fullsized":"2023-12-04T05:40:00.908703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:44:51.269794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681203+0000","last_change":"2023-12-04T05:39:59.718543+0000","last_active":"2023-12-04T05:40:00.681203+0000","last_peered":"2023-12-04T05:40:00.681203+0000","last_clean":"2023-12-04T05:40:00.681203+0000","last_became_active":"2023-12-04T05:39:59.717826+0000","last_became_peered":"2023-12-04T05:39:59.717826+0000","last_unstale":"2023-12-04T05:40:00.681203+0000","last_undegraded":"2023-12-04T05:40:00.681203+0000","last_fullsized":"2023-12-04T05:40:00.681203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:31:03.248114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.684087+0000","last_change":"2023-12-04T05:39:59.724278+0000","last_active":"2023-12-04T05:40:00.684087+0000","last_peered":"2023-12-04T05:40:00.684087+0000","last_clean":"2023-12-04T05:40:00.684087+0000","last_became_active":"2023-12-04T05:39:59.723384+0000","last_became_peered":"2023-12-04T05:39:59.723384+0000","last_unstale":"2023-12-04T05:40:00.684087+0000","last_undegraded":"2023-12-04T05:40:00.684087+0000","last_fullsized":"2023-12-04T05:40:00.684087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:07:20.576260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.679022+0000","last_change":"2023-12-04T05:39:59.711802+0000","last_active":"2023-12-04T05:40:00.679022+0000","last_peered":"2023-12-04T05:40:00.679022+0000","last_clean":"2023-12-04T05:40:00.679022+0000","last_became_active":"2023-12-04T05:39:59.711044+0000","last_became_peered":"2023-12-04T05:39:59.711044+0000","last_unstale":"2023-12-04T05:40:00.679022+0000","last_undegraded":"2023-12-04T05:40:00.679022+0000","last_fullsized":"2023-12-04T05:40:00.679022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:42:28.221010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503300+0000","last_change":"2023-12-04T05:39:59.725226+0000","last_active":"2023-12-04T05:40:01.503300+0000","last_peered":"2023-12-04T05:40:01.503300+0000","last_clean":"2023-12-04T05:40:01.503300+0000","last_became_active":"2023-12-04T05:39:59.724334+0000","last_became_peered":"2023-12-04T05:39:59.724334+0000","last_unstale":"2023-12-04T05:40:01.503300+0000","last_undegraded":"2023-12-04T05:40:01.503300+0000","last_fullsized":"2023-12-04T05:40:01.503300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:48:30.439285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501624+0000","last_change":"2023-12-04T05:39:59.729858+0000","last_active":"2023-12-04T05:40:01.501624+0000","last_peered":"2023-12-04T05:40:01.501624+0000","last_clean":"2023-12-04T05:40:01.501624+0000","last_became_active":"2023-12-04T05:39:59.728989+0000","last_became_peered":"2023-12-04T05:39:59.728989+0000","last_unstale":"2023-12-04T05:40:01.501624+0000","last_undegraded":"2023-12-04T05:40:01.501624+0000","last_fullsized":"2023-12-04T05:40:01.501624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:42:01.099743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501616+0000","last_change":"2023-12-04T05:39:59.724022+0000","last_active":"2023-12-04T05:40:01.501616+0000","last_peered":"2023-12-04T05:40:01.501616+0000","last_clean":"2023-12-04T05:40:01.501616+0000","last_became_active":"2023-12-04T05:39:59.723120+0000","last_became_peered":"2023-12-04T05:39:59.723120+0000","last_unstale":"2023-12-04T05:40:01.501616+0000","last_undegraded":"2023-12-04T05:40:01.501616+0000","last_fullsized":"2023-12-04T05:40:01.501616+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:13:50.048757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683322+0000","last_change":"2023-12-04T05:39:59.714806+0000","last_active":"2023-12-04T05:40:00.683322+0000","last_peered":"2023-12-04T05:40:00.683322+0000","last_clean":"2023-12-04T05:40:00.683322+0000","last_became_active":"2023-12-04T05:39:59.713917+0000","last_became_peered":"2023-12-04T05:39:59.713917+0000","last_unstale":"2023-12-04T05:40:00.683322+0000","last_undegraded":"2023-12-04T05:40:00.683322+0000","last_fullsized":"2023-12-04T05:40:00.683322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:18:03.128562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501966+0000","last_change":"2023-12-04T05:39:59.729098+0000","last_active":"2023-12-04T05:40:01.501966+0000","last_peered":"2023-12-04T05:40:01.501966+0000","last_clean":"2023-12-04T05:40:01.501966+0000","last_became_active":"2023-12-04T05:39:59.728449+0000","last_became_peered":"2023-12-04T05:39:59.728449+0000","last_unstale":"2023-12-04T05:40:01.501966+0000","last_undegraded":"2023-12-04T05:40:01.501966+0000","last_fullsized":"2023-12-04T05:40:01.501966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:25:12.098431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682551+0000","last_change":"2023-12-04T05:39:59.708300+0000","last_active":"2023-12-04T05:40:00.682551+0000","last_peered":"2023-12-04T05:40:00.682551+0000","last_clean":"2023-12-04T05:40:00.682551+0000","last_became_active":"2023-12-04T05:39:59.707530+0000","last_became_peered":"2023-12-04T05:39:59.707530+0000","last_unstale":"2023-12-04T05:40:00.682551+0000","last_undegraded":"2023-12-04T05:40:00.682551+0000","last_fullsized":"2023-12-04T05:40:00.682551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:11:40.053312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502562+0000","last_change":"2023-12-04T05:39:59.729406+0000","last_active":"2023-12-04T05:40:01.502562+0000","last_peered":"2023-12-04T05:40:01.502562+0000","last_clean":"2023-12-04T05:40:01.502562+0000","last_became_active":"2023-12-04T05:39:59.728677+0000","last_became_peered":"2023-12-04T05:39:59.728677+0000","last_unstale":"2023-12-04T05:40:01.502562+0000","last_undegraded":"2023-12-04T05:40:01.502562+0000","last_fullsized":"2023-12-04T05:40:01.502562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:03:26.954387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501922+0000","last_change":"2023-12-04T05:39:59.733628+0000","last_active":"2023-12-04T05:40:01.501922+0000","last_peered":"2023-12-04T05:40:01.501922+0000","last_clean":"2023-12-04T05:40:01.501922+0000","last_became_active":"2023-12-04T05:39:59.726137+0000","last_became_peered":"2023-12-04T05:39:59.726137+0000","last_unstale":"2023-12-04T05:40:01.501922+0000","last_undegraded":"2023-12-04T05:40:01.501922+0000","last_fullsized":"2023-12-04T05:40:01.501922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:49:17.630085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908788+0000","last_change":"2023-12-04T05:39:59.727169+0000","last_active":"2023-12-04T05:40:00.908788+0000","last_peered":"2023-12-04T05:40:00.908788+0000","last_clean":"2023-12-04T05:40:00.908788+0000","last_became_active":"2023-12-04T05:39:59.725378+0000","last_became_peered":"2023-12-04T05:39:59.725378+0000","last_unstale":"2023-12-04T05:40:00.908788+0000","last_undegraded":"2023-12-04T05:40:00.908788+0000","last_fullsized":"2023-12-04T05:40:00.908788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:23:21.007397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502427+0000","last_change":"2023-12-04T05:39:59.729516+0000","last_active":"2023-12-04T05:40:01.502427+0000","last_peered":"2023-12-04T05:40:01.502427+0000","last_clean":"2023-12-04T05:40:01.502427+0000","last_became_active":"2023-12-04T05:39:59.728835+0000","last_became_peered":"2023-12-04T05:39:59.728835+0000","last_unstale":"2023-12-04T05:40:01.502427+0000","last_undegraded":"2023-12-04T05:40:01.502427+0000","last_fullsized":"2023-12-04T05:40:01.502427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:37:37.490362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501933+0000","last_change":"2023-12-04T05:39:59.724592+0000","last_active":"2023-12-04T05:40:01.501933+0000","last_peered":"2023-12-04T05:40:01.501933+0000","last_clean":"2023-12-04T05:40:01.501933+0000","last_became_active":"2023-12-04T05:39:59.723876+0000","last_became_peered":"2023-12-04T05:39:59.723876+0000","last_unstale":"2023-12-04T05:40:01.501933+0000","last_undegraded":"2023-12-04T05:40:01.501933+0000","last_fullsized":"2023-12-04T05:40:01.501933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:09:26.625728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683703+0000","last_change":"2023-12-04T05:39:59.726540+0000","last_active":"2023-12-04T05:40:00.683703+0000","last_peered":"2023-12-04T05:40:00.683703+0000","last_clean":"2023-12-04T05:40:00.683703+0000","last_became_active":"2023-12-04T05:39:59.725781+0000","last_became_peered":"2023-12-04T05:39:59.725781+0000","last_unstale":"2023-12-04T05:40:00.683703+0000","last_undegraded":"2023-12-04T05:40:00.683703+0000","last_fullsized":"2023-12-04T05:40:00.683703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:59:22.013410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.678751+0000","last_change":"2023-12-04T05:39:59.708376+0000","last_active":"2023-12-04T05:40:00.678751+0000","last_peered":"2023-12-04T05:40:00.678751+0000","last_clean":"2023-12-04T05:40:00.678751+0000","last_became_active":"2023-12-04T05:39:59.707604+0000","last_became_peered":"2023-12-04T05:39:59.707604+0000","last_unstale":"2023-12-04T05:40:00.678751+0000","last_undegraded":"2023-12-04T05:40:00.678751+0000","last_fullsized":"2023-12-04T05:40:00.678751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:00:23.385619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501476+0000","last_change":"2023-12-04T05:39:59.720510+0000","last_active":"2023-12-04T05:40:01.501476+0000","last_peered":"2023-12-04T05:40:01.501476+0000","last_clean":"2023-12-04T05:40:01.501476+0000","last_became_active":"2023-12-04T05:39:59.719704+0000","last_became_peered":"2023-12-04T05:39:59.719704+0000","last_unstale":"2023-12-04T05:40:01.501476+0000","last_undegraded":"2023-12-04T05:40:01.501476+0000","last_fullsized":"2023-12-04T05:40:01.501476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:39:15.714817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681790+0000","last_change":"2023-12-04T05:39:59.711839+0000","last_active":"2023-12-04T05:40:00.681790+0000","last_peered":"2023-12-04T05:40:00.681790+0000","last_clean":"2023-12-04T05:40:00.681790+0000","last_became_active":"2023-12-04T05:39:59.711113+0000","last_became_peered":"2023-12-04T05:39:59.711113+0000","last_unstale":"2023-12-04T05:40:00.681790+0000","last_undegraded":"2023-12-04T05:40:00.681790+0000","last_fullsized":"2023-12-04T05:40:00.681790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:21:17.569525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.529158+0000","last_change":"2023-12-04T05:39:58.750554+0000","last_active":"2023-12-04T05:40:01.529158+0000","last_peered":"2023-12-04T05:40:01.529158+0000","last_clean":"2023-12-04T05:40:01.529158+0000","last_became_active":"2023-12-04T05:39:58.750227+0000","last_became_peered":"2023-12-04T05:39:58.750227+0000","last_unstale":"2023-12-04T05:40:01.529158+0000","last_undegraded":"2023-12-04T05:40:01.529158+0000","last_fullsized":"2023-12-04T05:40:01.529158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:13:23.997255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502276+0000","last_change":"2023-12-04T05:39:59.731582+0000","last_active":"2023-12-04T05:40:01.502276+0000","last_peered":"2023-12-04T05:40:01.502276+0000","last_clean":"2023-12-04T05:40:01.502276+0000","last_became_active":"2023-12-04T05:39:59.730822+0000","last_became_peered":"2023-12-04T05:39:59.730822+0000","last_unstale":"2023-12-04T05:40:01.502276+0000","last_undegraded":"2023-12-04T05:40:01.502276+0000","last_fullsized":"2023-12-04T05:40:01.502276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:10:55.996104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.524401+0000","last_change":"2023-12-04T05:39:58.740011+0000","last_active":"2023-12-04T05:40:01.524401+0000","last_peered":"2023-12-04T05:40:01.524401+0000","last_clean":"2023-12-04T05:40:01.524401+0000","last_became_active":"2023-12-04T05:39:58.739395+0000","last_became_peered":"2023-12-04T05:39:58.739395+0000","last_unstale":"2023-12-04T05:40:01.524401+0000","last_undegraded":"2023-12-04T05:40:01.524401+0000","last_fullsized":"2023-12-04T05:40:01.524401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:17:48.107311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502182+0000","last_change":"2023-12-04T05:39:58.730186+0000","last_active":"2023-12-04T05:40:01.502182+0000","last_peered":"2023-12-04T05:40:01.502182+0000","last_clean":"2023-12-04T05:40:01.502182+0000","last_became_active":"2023-12-04T05:39:58.729411+0000","last_became_peered":"2023-12-04T05:39:58.729411+0000","last_unstale":"2023-12-04T05:40:01.502182+0000","last_undegraded":"2023-12-04T05:40:01.502182+0000","last_fullsized":"2023-12-04T05:40:01.502182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:39:05.463010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502530+0000","last_change":"2023-12-04T05:39:59.715596+0000","last_active":"2023-12-04T05:40:01.502530+0000","last_peered":"2023-12-04T05:40:01.502530+0000","last_clean":"2023-12-04T05:40:01.502530+0000","last_became_active":"2023-12-04T05:39:59.714842+0000","last_became_peered":"2023-12-04T05:39:59.714842+0000","last_unstale":"2023-12-04T05:40:01.502530+0000","last_undegraded":"2023-12-04T05:40:01.502530+0000","last_fullsized":"2023-12-04T05:40:01.502530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:51:10.874053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682760+0000","last_change":"2023-12-04T05:39:58.722113+0000","last_active":"2023-12-04T05:40:00.682760+0000","last_peered":"2023-12-04T05:40:00.682760+0000","last_clean":"2023-12-04T05:40:00.682760+0000","last_became_active":"2023-12-04T05:39:58.721346+0000","last_became_peered":"2023-12-04T05:39:58.721346+0000","last_unstale":"2023-12-04T05:40:00.682760+0000","last_undegraded":"2023-12-04T05:40:00.682760+0000","last_fullsized":"2023-12-04T05:40:00.682760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:30:11.578047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908461+0000","last_change":"2023-12-04T05:39:59.708575+0000","last_active":"2023-12-04T05:40:00.908461+0000","last_peered":"2023-12-04T05:40:00.908461+0000","last_clean":"2023-12-04T05:40:00.908461+0000","last_became_active":"2023-12-04T05:39:59.707753+0000","last_became_peered":"2023-12-04T05:39:59.707753+0000","last_unstale":"2023-12-04T05:40:00.908461+0000","last_undegraded":"2023-12-04T05:40:00.908461+0000","last_fullsized":"2023-12-04T05:40:00.908461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:47:58.349368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501888+0000","last_change":"2023-12-04T05:39:58.734739+0000","last_active":"2023-12-04T05:40:01.501888+0000","last_peered":"2023-12-04T05:40:01.501888+0000","last_clean":"2023-12-04T05:40:01.501888+0000","last_became_active":"2023-12-04T05:39:58.733897+0000","last_became_peered":"2023-12-04T05:39:58.733897+0000","last_unstale":"2023-12-04T05:40:01.501888+0000","last_undegraded":"2023-12-04T05:40:01.501888+0000","last_fullsized":"2023-12-04T05:40:01.501888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:52:15.157364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683392+0000","last_change":"2023-12-04T05:39:59.715796+0000","last_active":"2023-12-04T05:40:00.683392+0000","last_peered":"2023-12-04T05:40:00.683392+0000","last_clean":"2023-12-04T05:40:00.683392+0000","last_became_active":"2023-12-04T05:39:59.714951+0000","last_became_peered":"2023-12-04T05:39:59.714951+0000","last_unstale":"2023-12-04T05:40:00.683392+0000","last_undegraded":"2023-12-04T05:40:00.683392+0000","last_fullsized":"2023-12-04T05:40:00.683392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:31:12.475136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683603+0000","last_change":"2023-12-04T05:39:58.725387+0000","last_active":"2023-12-04T05:40:00.683603+0000","last_peered":"2023-12-04T05:40:00.683603+0000","last_clean":"2023-12-04T05:40:00.683603+0000","last_became_active":"2023-12-04T05:39:58.724672+0000","last_became_peered":"2023-12-04T05:39:58.724672+0000","last_unstale":"2023-12-04T05:40:00.683603+0000","last_undegraded":"2023-12-04T05:40:00.683603+0000","last_fullsized":"2023-12-04T05:40:00.683603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:19:43.112172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503316+0000","last_change":"2023-12-04T05:39:59.718789+0000","last_active":"2023-12-04T05:40:01.503316+0000","last_peered":"2023-12-04T05:40:01.503316+0000","last_clean":"2023-12-04T05:40:01.503316+0000","last_became_active":"2023-12-04T05:39:59.717927+0000","last_became_peered":"2023-12-04T05:39:59.717927+0000","last_unstale":"2023-12-04T05:40:01.503316+0000","last_undegraded":"2023-12-04T05:40:01.503316+0000","last_fullsized":"2023-12-04T05:40:01.503316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:43:08.771461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.967861+0000","last_change":"2023-12-04T05:39:58.731710+0000","last_active":"2023-12-04T05:40:04.967861+0000","last_peered":"2023-12-04T05:40:04.967861+0000","last_clean":"2023-12-04T05:40:04.967861+0000","last_became_active":"2023-12-04T05:39:58.731373+0000","last_became_peered":"2023-12-04T05:39:58.731373+0000","last_unstale":"2023-12-04T05:40:04.967861+0000","last_undegraded":"2023-12-04T05:40:04.967861+0000","last_fullsized":"2023-12-04T05:40:04.967861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:03:37.899195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682594+0000","last_change":"2023-12-04T05:39:59.725444+0000","last_active":"2023-12-04T05:40:00.682594+0000","last_peered":"2023-12-04T05:40:00.682594+0000","last_clean":"2023-12-04T05:40:00.682594+0000","last_became_active":"2023-12-04T05:39:59.724620+0000","last_became_peered":"2023-12-04T05:39:59.724620+0000","last_unstale":"2023-12-04T05:40:00.682594+0000","last_undegraded":"2023-12-04T05:40:00.682594+0000","last_fullsized":"2023-12-04T05:40:00.682594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:11:17.092060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.781096+0000","last_change":"2023-12-04T05:39:58.730135+0000","last_active":"2023-12-04T05:40:04.781096+0000","last_peered":"2023-12-04T05:40:04.781096+0000","last_clean":"2023-12-04T05:40:04.781096+0000","last_became_active":"2023-12-04T05:39:58.729295+0000","last_became_peered":"2023-12-04T05:39:58.729295+0000","last_unstale":"2023-12-04T05:40:04.781096+0000","last_undegraded":"2023-12-04T05:40:04.781096+0000","last_fullsized":"2023-12-04T05:40:04.781096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:24:42.912063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502658+0000","last_change":"2023-12-04T05:39:59.731327+0000","last_active":"2023-12-04T05:40:01.502658+0000","last_peered":"2023-12-04T05:40:01.502658+0000","last_clean":"2023-12-04T05:40:01.502658+0000","last_became_active":"2023-12-04T05:39:59.730427+0000","last_became_peered":"2023-12-04T05:39:59.730427+0000","last_unstale":"2023-12-04T05:40:01.502658+0000","last_undegraded":"2023-12-04T05:40:01.502658+0000","last_fullsized":"2023-12-04T05:40:01.502658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:34:45.211951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.767631+0000","last_change":"2023-12-04T05:39:58.711690+0000","last_active":"2023-12-04T05:40:04.767631+0000","last_peered":"2023-12-04T05:40:04.767631+0000","last_clean":"2023-12-04T05:40:04.767631+0000","last_became_active":"2023-12-04T05:39:58.711305+0000","last_became_peered":"2023-12-04T05:39:58.711305+0000","last_unstale":"2023-12-04T05:40:04.767631+0000","last_undegraded":"2023-12-04T05:40:04.767631+0000","last_fullsized":"2023-12-04T05:40:04.767631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:33:02.159265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501048+0000","last_change":"2023-12-04T05:39:59.702650+0000","last_active":"2023-12-04T05:40:01.501048+0000","last_peered":"2023-12-04T05:40:01.501048+0000","last_clean":"2023-12-04T05:40:01.501048+0000","last_became_active":"2023-12-04T05:39:59.701827+0000","last_became_peered":"2023-12-04T05:39:59.701827+0000","last_unstale":"2023-12-04T05:40:01.501048+0000","last_undegraded":"2023-12-04T05:40:01.501048+0000","last_fullsized":"2023-12-04T05:40:01.501048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:46:45.006062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683932+0000","last_change":"2023-12-04T05:39:58.729371+0000","last_active":"2023-12-04T05:40:00.683932+0000","last_peered":"2023-12-04T05:40:00.683932+0000","last_clean":"2023-12-04T05:40:00.683932+0000","last_became_active":"2023-12-04T05:39:58.727781+0000","last_became_peered":"2023-12-04T05:39:58.727781+0000","last_unstale":"2023-12-04T05:40:00.683932+0000","last_undegraded":"2023-12-04T05:40:00.683932+0000","last_fullsized":"2023-12-04T05:40:00.683932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:37:29.707707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502646+0000","last_change":"2023-12-04T05:39:59.742080+0000","last_active":"2023-12-04T05:40:01.502646+0000","last_peered":"2023-12-04T05:40:01.502646+0000","last_clean":"2023-12-04T05:40:01.502646+0000","last_became_active":"2023-12-04T05:39:59.741440+0000","last_became_peered":"2023-12-04T05:39:59.741440+0000","last_unstale":"2023-12-04T05:40:01.502646+0000","last_undegraded":"2023-12-04T05:40:01.502646+0000","last_fullsized":"2023-12-04T05:40:01.502646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:29:21.379583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.500870+0000","last_change":"2023-12-04T05:39:58.737926+0000","last_active":"2023-12-04T05:40:01.500870+0000","last_peered":"2023-12-04T05:40:01.500870+0000","last_clean":"2023-12-04T05:40:01.500870+0000","last_became_active":"2023-12-04T05:39:58.737403+0000","last_became_peered":"2023-12-04T05:39:58.737403+0000","last_unstale":"2023-12-04T05:40:01.500870+0000","last_undegraded":"2023-12-04T05:40:01.500870+0000","last_fullsized":"2023-12-04T05:40:01.500870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:10:59.035282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501005+0000","last_change":"2023-12-04T05:39:59.710310+0000","last_active":"2023-12-04T05:40:01.501005+0000","last_peered":"2023-12-04T05:40:01.501005+0000","last_clean":"2023-12-04T05:40:01.501005+0000","last_became_active":"2023-12-04T05:39:59.709497+0000","last_became_peered":"2023-12-04T05:39:59.709497+0000","last_unstale":"2023-12-04T05:40:01.501005+0000","last_undegraded":"2023-12-04T05:40:01.501005+0000","last_fullsized":"2023-12-04T05:40:01.501005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:43:43.708311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.779639+0000","last_change":"2023-12-04T05:39:58.728046+0000","last_active":"2023-12-04T05:40:04.779639+0000","last_peered":"2023-12-04T05:40:04.779639+0000","last_clean":"2023-12-04T05:40:04.779639+0000","last_became_active":"2023-12-04T05:39:58.726747+0000","last_became_peered":"2023-12-04T05:39:58.726747+0000","last_unstale":"2023-12-04T05:40:04.779639+0000","last_undegraded":"2023-12-04T05:40:04.779639+0000","last_fullsized":"2023-12-04T05:40:04.779639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:45:11.024395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501689+0000","last_change":"2023-12-04T05:39:59.741770+0000","last_active":"2023-12-04T05:40:01.501689+0000","last_peered":"2023-12-04T05:40:01.501689+0000","last_clean":"2023-12-04T05:40:01.501689+0000","last_became_active":"2023-12-04T05:39:59.741091+0000","last_became_peered":"2023-12-04T05:39:59.741091+0000","last_unstale":"2023-12-04T05:40:01.501689+0000","last_undegraded":"2023-12-04T05:40:01.501689+0000","last_fullsized":"2023-12-04T05:40:01.501689+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:59:20.747612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.524182+0000","last_change":"2023-12-04T05:39:58.712970+0000","last_active":"2023-12-04T05:40:01.524182+0000","last_peered":"2023-12-04T05:40:01.524182+0000","last_clean":"2023-12-04T05:40:01.524182+0000","last_became_active":"2023-12-04T05:39:58.711759+0000","last_became_peered":"2023-12-04T05:39:58.711759+0000","last_unstale":"2023-12-04T05:40:01.524182+0000","last_undegraded":"2023-12-04T05:40:01.524182+0000","last_fullsized":"2023-12-04T05:40:01.524182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:02:00.957563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503407+0000","last_change":"2023-12-04T05:39:59.742244+0000","last_active":"2023-12-04T05:40:01.503407+0000","last_peered":"2023-12-04T05:40:01.503407+0000","last_clean":"2023-12-04T05:40:01.503407+0000","last_became_active":"2023-12-04T05:39:59.741379+0000","last_became_peered":"2023-12-04T05:39:59.741379+0000","last_unstale":"2023-12-04T05:40:01.503407+0000","last_undegraded":"2023-12-04T05:40:01.503407+0000","last_fullsized":"2023-12-04T05:40:01.503407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:07:57.685895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.769288+0000","last_change":"2023-12-04T05:39:58.729723+0000","last_active":"2023-12-04T05:40:04.769288+0000","last_peered":"2023-12-04T05:40:04.769288+0000","last_clean":"2023-12-04T05:40:04.769288+0000","last_became_active":"2023-12-04T05:39:58.729410+0000","last_became_peered":"2023-12-04T05:39:58.729410+0000","last_unstale":"2023-12-04T05:40:04.769288+0000","last_undegraded":"2023-12-04T05:40:04.769288+0000","last_fullsized":"2023-12-04T05:40:04.769288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:59:10.799774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503914+0000","last_change":"2023-12-04T05:39:59.741718+0000","last_active":"2023-12-04T05:40:01.503914+0000","last_peered":"2023-12-04T05:40:01.503914+0000","last_clean":"2023-12-04T05:40:01.503914+0000","last_became_active":"2023-12-04T05:39:59.741101+0000","last_became_peered":"2023-12-04T05:39:59.741101+0000","last_unstale":"2023-12-04T05:40:01.503914+0000","last_undegraded":"2023-12-04T05:40:01.503914+0000","last_fullsized":"2023-12-04T05:40:01.503914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:41:03.969161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.521601+0000","last_change":"2023-12-04T05:39:58.737224+0000","last_active":"2023-12-04T05:40:01.521601+0000","last_peered":"2023-12-04T05:40:01.521601+0000","last_clean":"2023-12-04T05:40:01.521601+0000","last_became_active":"2023-12-04T05:39:58.736753+0000","last_became_peered":"2023-12-04T05:39:58.736753+0000","last_unstale":"2023-12-04T05:40:01.521601+0000","last_undegraded":"2023-12-04T05:40:01.521601+0000","last_fullsized":"2023-12-04T05:40:01.521601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:33:34.027796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683405+0000","last_change":"2023-12-04T05:39:59.726776+0000","last_active":"2023-12-04T05:40:00.683405+0000","last_peered":"2023-12-04T05:40:00.683405+0000","last_clean":"2023-12-04T05:40:00.683405+0000","last_became_active":"2023-12-04T05:39:59.725991+0000","last_became_peered":"2023-12-04T05:39:59.725991+0000","last_unstale":"2023-12-04T05:40:00.683405+0000","last_undegraded":"2023-12-04T05:40:00.683405+0000","last_fullsized":"2023-12-04T05:40:00.683405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:36:02.168898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.678822+0000","last_change":"2023-12-04T05:39:58.732942+0000","last_active":"2023-12-04T05:40:00.678822+0000","last_peered":"2023-12-04T05:40:00.678822+0000","last_clean":"2023-12-04T05:40:00.678822+0000","last_became_active":"2023-12-04T05:39:58.732401+0000","last_became_peered":"2023-12-04T05:39:58.732401+0000","last_unstale":"2023-12-04T05:40:00.678822+0000","last_undegraded":"2023-12-04T05:40:00.678822+0000","last_fullsized":"2023-12-04T05:40:00.678822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:23:13.676061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503215+0000","last_change":"2023-12-04T05:39:59.730768+0000","last_active":"2023-12-04T05:40:01.503215+0000","last_peered":"2023-12-04T05:40:01.503215+0000","last_clean":"2023-12-04T05:40:01.503215+0000","last_became_active":"2023-12-04T05:39:59.730207+0000","last_became_peered":"2023-12-04T05:39:59.730207+0000","last_unstale":"2023-12-04T05:40:01.503215+0000","last_undegraded":"2023-12-04T05:40:01.503215+0000","last_fullsized":"2023-12-04T05:40:01.503215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:25:19.473188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T05:39:59.705581+0000","last_change":"2023-12-04T05:39:49.314208+0000","last_active":"2023-12-04T05:39:59.705581+0000","last_peered":"2023-12-04T05:39:59.705581+0000","last_clean":"2023-12-04T05:39:59.705581+0000","last_became_active":"2023-12-04T05:39:49.312262+0000","last_became_peered":"2023-12-04T05:39:49.312262+0000","last_unstale":"2023-12-04T05:39:59.705581+0000","last_undegraded":"2023-12-04T05:39:59.705581+0000","last_fullsized":"2023-12-04T05:39:59.705581+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:48.242612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:48.242612+0000","last_clean_scrub_stamp":"2023-12-04T05:39:48.242612+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:19:26.511345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682745+0000","last_change":"2023-12-04T05:39:58.725538+0000","last_active":"2023-12-04T05:40:00.682745+0000","last_peered":"2023-12-04T05:40:00.682745+0000","last_clean":"2023-12-04T05:40:00.682745+0000","last_became_active":"2023-12-04T05:39:58.724839+0000","last_became_peered":"2023-12-04T05:39:58.724839+0000","last_unstale":"2023-12-04T05:40:00.682745+0000","last_undegraded":"2023-12-04T05:40:00.682745+0000","last_fullsized":"2023-12-04T05:40:00.682745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:34:48.521117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501800+0000","last_change":"2023-12-04T05:39:59.719475+0000","last_active":"2023-12-04T05:40:01.501800+0000","last_peered":"2023-12-04T05:40:01.501800+0000","last_clean":"2023-12-04T05:40:01.501800+0000","last_became_active":"2023-12-04T05:39:59.718727+0000","last_became_peered":"2023-12-04T05:39:59.718727+0000","last_unstale":"2023-12-04T05:40:01.501800+0000","last_undegraded":"2023-12-04T05:40:01.501800+0000","last_fullsized":"2023-12-04T05:40:01.501800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:53:35.787086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501396+0000","last_change":"2023-12-04T05:39:58.751135+0000","last_active":"2023-12-04T05:40:01.501396+0000","last_peered":"2023-12-04T05:40:01.501396+0000","last_clean":"2023-12-04T05:40:01.501396+0000","last_became_active":"2023-12-04T05:39:58.750144+0000","last_became_peered":"2023-12-04T05:39:58.750144+0000","last_unstale":"2023-12-04T05:40:01.501396+0000","last_undegraded":"2023-12-04T05:40:01.501396+0000","last_fullsized":"2023-12-04T05:40:01.501396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:31:32.599659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.500789+0000","last_change":"2023-12-04T05:39:59.729585+0000","last_active":"2023-12-04T05:40:01.500789+0000","last_peered":"2023-12-04T05:40:01.500789+0000","last_clean":"2023-12-04T05:40:01.500789+0000","last_became_active":"2023-12-04T05:39:59.728381+0000","last_became_peered":"2023-12-04T05:39:59.728381+0000","last_unstale":"2023-12-04T05:40:01.500789+0000","last_undegraded":"2023-12-04T05:40:01.500789+0000","last_fullsized":"2023-12-04T05:40:01.500789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:35:11.920755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.678836+0000","last_change":"2023-12-04T05:39:58.731429+0000","last_active":"2023-12-04T05:40:00.678836+0000","last_peered":"2023-12-04T05:40:00.678836+0000","last_clean":"2023-12-04T05:40:00.678836+0000","last_became_active":"2023-12-04T05:39:58.731049+0000","last_became_peered":"2023-12-04T05:39:58.731049+0000","last_unstale":"2023-12-04T05:40:00.678836+0000","last_undegraded":"2023-12-04T05:40:00.678836+0000","last_fullsized":"2023-12-04T05:40:00.678836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:27:16.863460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908531+0000","last_change":"2023-12-04T05:39:59.701324+0000","last_active":"2023-12-04T05:40:00.908531+0000","last_peered":"2023-12-04T05:40:00.908531+0000","last_clean":"2023-12-04T05:40:00.908531+0000","last_became_active":"2023-12-04T05:39:59.700586+0000","last_became_peered":"2023-12-04T05:39:59.700586+0000","last_unstale":"2023-12-04T05:40:00.908531+0000","last_undegraded":"2023-12-04T05:40:00.908531+0000","last_fullsized":"2023-12-04T05:40:00.908531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:37:17.301688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.684152+0000","last_change":"2023-12-04T05:39:58.729635+0000","last_active":"2023-12-04T05:40:00.684152+0000","last_peered":"2023-12-04T05:40:00.684152+0000","last_clean":"2023-12-04T05:40:00.684152+0000","last_became_active":"2023-12-04T05:39:58.728927+0000","last_became_peered":"2023-12-04T05:39:58.728927+0000","last_unstale":"2023-12-04T05:40:00.684152+0000","last_undegraded":"2023-12-04T05:40:00.684152+0000","last_fullsized":"2023-12-04T05:40:00.684152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:58:46.851065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681790+0000","last_change":"2023-12-04T05:39:59.702840+0000","last_active":"2023-12-04T05:40:00.681790+0000","last_peered":"2023-12-04T05:40:00.681790+0000","last_clean":"2023-12-04T05:40:00.681790+0000","last_became_active":"2023-12-04T05:39:59.701652+0000","last_became_peered":"2023-12-04T05:39:59.701652+0000","last_unstale":"2023-12-04T05:40:00.681790+0000","last_undegraded":"2023-12-04T05:40:00.681790+0000","last_fullsized":"2023-12-04T05:40:00.681790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:44:14.733182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908769+0000","last_change":"2023-12-04T05:39:58.721834+0000","last_active":"2023-12-04T05:40:00.908769+0000","last_peered":"2023-12-04T05:40:00.908769+0000","last_clean":"2023-12-04T05:40:00.908769+0000","last_became_active":"2023-12-04T05:39:58.721451+0000","last_became_peered":"2023-12-04T05:39:58.721451+0000","last_unstale":"2023-12-04T05:40:00.908769+0000","last_undegraded":"2023-12-04T05:40:00.908769+0000","last_fullsized":"2023-12-04T05:40:00.908769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:15:52.065853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.684129+0000","last_change":"2023-12-04T05:39:59.731053+0000","last_active":"2023-12-04T05:40:00.684129+0000","last_peered":"2023-12-04T05:40:00.684129+0000","last_clean":"2023-12-04T05:40:00.684129+0000","last_became_active":"2023-12-04T05:39:59.729732+0000","last_became_peered":"2023-12-04T05:39:59.729732+0000","last_unstale":"2023-12-04T05:40:00.684129+0000","last_undegraded":"2023-12-04T05:40:00.684129+0000","last_fullsized":"2023-12-04T05:40:00.684129+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:57:38.581311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503452+0000","last_change":"2023-12-04T05:39:58.751126+0000","last_active":"2023-12-04T05:40:01.503452+0000","last_peered":"2023-12-04T05:40:01.503452+0000","last_clean":"2023-12-04T05:40:01.503452+0000","last_became_active":"2023-12-04T05:39:58.750496+0000","last_became_peered":"2023-12-04T05:39:58.750496+0000","last_unstale":"2023-12-04T05:40:01.503452+0000","last_undegraded":"2023-12-04T05:40:01.503452+0000","last_fullsized":"2023-12-04T05:40:01.503452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:09:22.333993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908579+0000","last_change":"2023-12-04T05:39:59.700032+0000","last_active":"2023-12-04T05:40:00.908579+0000","last_peered":"2023-12-04T05:40:00.908579+0000","last_clean":"2023-12-04T05:40:00.908579+0000","last_became_active":"2023-12-04T05:39:59.699488+0000","last_became_peered":"2023-12-04T05:39:59.699488+0000","last_unstale":"2023-12-04T05:40:00.908579+0000","last_undegraded":"2023-12-04T05:40:00.908579+0000","last_fullsized":"2023-12-04T05:40:00.908579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:59:57.153486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.909014+0000","last_change":"2023-12-04T05:39:58.723126+0000","last_active":"2023-12-04T05:40:00.909014+0000","last_peered":"2023-12-04T05:40:00.909014+0000","last_clean":"2023-12-04T05:40:00.909014+0000","last_became_active":"2023-12-04T05:39:58.722766+0000","last_became_peered":"2023-12-04T05:39:58.722766+0000","last_unstale":"2023-12-04T05:40:00.909014+0000","last_undegraded":"2023-12-04T05:40:00.909014+0000","last_fullsized":"2023-12-04T05:40:00.909014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:28:47.171345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502918+0000","last_change":"2023-12-04T05:39:59.724653+0000","last_active":"2023-12-04T05:40:01.502918+0000","last_peered":"2023-12-04T05:40:01.502918+0000","last_clean":"2023-12-04T05:40:01.502918+0000","last_became_active":"2023-12-04T05:39:59.723945+0000","last_became_peered":"2023-12-04T05:39:59.723945+0000","last_unstale":"2023-12-04T05:40:01.502918+0000","last_undegraded":"2023-12-04T05:40:01.502918+0000","last_fullsized":"2023-12-04T05:40:01.502918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:13:56.812140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.772912+0000","last_change":"2023-12-04T05:39:58.746966+0000","last_active":"2023-12-04T05:40:04.772912+0000","last_peered":"2023-12-04T05:40:04.772912+0000","last_clean":"2023-12-04T05:40:04.772912+0000","last_became_active":"2023-12-04T05:39:58.746261+0000","last_became_peered":"2023-12-04T05:39:58.746261+0000","last_unstale":"2023-12-04T05:40:04.772912+0000","last_undegraded":"2023-12-04T05:40:04.772912+0000","last_fullsized":"2023-12-04T05:40:04.772912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:57:46.899291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503381+0000","last_change":"2023-12-04T05:39:59.713259+0000","last_active":"2023-12-04T05:40:01.503381+0000","last_peered":"2023-12-04T05:40:01.503381+0000","last_clean":"2023-12-04T05:40:01.503381+0000","last_became_active":"2023-12-04T05:39:59.712523+0000","last_became_peered":"2023-12-04T05:39:59.712523+0000","last_unstale":"2023-12-04T05:40:01.503381+0000","last_undegraded":"2023-12-04T05:40:01.503381+0000","last_fullsized":"2023-12-04T05:40:01.503381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:41:10.001755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501174+0000","last_change":"2023-12-04T05:39:58.750282+0000","last_active":"2023-12-04T05:40:01.501174+0000","last_peered":"2023-12-04T05:40:01.501174+0000","last_clean":"2023-12-04T05:40:01.501174+0000","last_became_active":"2023-12-04T05:39:58.749651+0000","last_became_peered":"2023-12-04T05:39:58.749651+0000","last_unstale":"2023-12-04T05:40:01.501174+0000","last_undegraded":"2023-12-04T05:40:01.501174+0000","last_fullsized":"2023-12-04T05:40:01.501174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:40:55.086649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681687+0000","last_change":"2023-12-04T05:39:59.704667+0000","last_active":"2023-12-04T05:40:00.681687+0000","last_peered":"2023-12-04T05:40:00.681687+0000","last_clean":"2023-12-04T05:40:00.681687+0000","last_became_active":"2023-12-04T05:39:59.703943+0000","last_became_peered":"2023-12-04T05:39:59.703943+0000","last_unstale":"2023-12-04T05:40:00.681687+0000","last_undegraded":"2023-12-04T05:40:00.681687+0000","last_fullsized":"2023-12-04T05:40:00.681687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:31:53.890532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908698+0000","last_change":"2023-12-04T05:39:58.745277+0000","last_active":"2023-12-04T05:40:00.908698+0000","last_peered":"2023-12-04T05:40:00.908698+0000","last_clean":"2023-12-04T05:40:00.908698+0000","last_became_active":"2023-12-04T05:39:58.744842+0000","last_became_peered":"2023-12-04T05:39:58.744842+0000","last_unstale":"2023-12-04T05:40:00.908698+0000","last_undegraded":"2023-12-04T05:40:00.908698+0000","last_fullsized":"2023-12-04T05:40:00.908698+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:50:21.084855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502964+0000","last_change":"2023-12-04T05:39:59.714217+0000","last_active":"2023-12-04T05:40:01.502964+0000","last_peered":"2023-12-04T05:40:01.502964+0000","last_clean":"2023-12-04T05:40:01.502964+0000","last_became_active":"2023-12-04T05:39:59.713438+0000","last_became_peered":"2023-12-04T05:39:59.713438+0000","last_unstale":"2023-12-04T05:40:01.502964+0000","last_undegraded":"2023-12-04T05:40:01.502964+0000","last_fullsized":"2023-12-04T05:40:01.502964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:37:30.790544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908612+0000","last_change":"2023-12-04T05:39:58.722048+0000","last_active":"2023-12-04T05:40:00.908612+0000","last_peered":"2023-12-04T05:40:00.908612+0000","last_clean":"2023-12-04T05:40:00.908612+0000","last_became_active":"2023-12-04T05:39:58.721782+0000","last_became_peered":"2023-12-04T05:39:58.721782+0000","last_unstale":"2023-12-04T05:40:00.908612+0000","last_undegraded":"2023-12-04T05:40:00.908612+0000","last_fullsized":"2023-12-04T05:40:00.908612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:38:16.023373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502222+0000","last_change":"2023-12-04T05:39:59.715313+0000","last_active":"2023-12-04T05:40:01.502222+0000","last_peered":"2023-12-04T05:40:01.502222+0000","last_clean":"2023-12-04T05:40:01.502222+0000","last_became_active":"2023-12-04T05:39:59.714430+0000","last_became_peered":"2023-12-04T05:39:59.714430+0000","last_unstale":"2023-12-04T05:40:01.502222+0000","last_undegraded":"2023-12-04T05:40:01.502222+0000","last_fullsized":"2023-12-04T05:40:01.502222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:10:01.533077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.769466+0000","last_change":"2023-12-04T05:39:58.750921+0000","last_active":"2023-12-04T05:40:04.769466+0000","last_peered":"2023-12-04T05:40:04.769466+0000","last_clean":"2023-12-04T05:40:04.769466+0000","last_became_active":"2023-12-04T05:39:58.750288+0000","last_became_peered":"2023-12-04T05:39:58.750288+0000","last_unstale":"2023-12-04T05:40:04.769466+0000","last_undegraded":"2023-12-04T05:40:04.769466+0000","last_fullsized":"2023-12-04T05:40:04.769466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:04:26.373696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501279+0000","last_change":"2023-12-04T05:39:59.729260+0000","last_active":"2023-12-04T05:40:01.501279+0000","last_peered":"2023-12-04T05:40:01.501279+0000","last_clean":"2023-12-04T05:40:01.501279+0000","last_became_active":"2023-12-04T05:39:59.728639+0000","last_became_peered":"2023-12-04T05:39:59.728639+0000","last_unstale":"2023-12-04T05:40:01.501279+0000","last_undegraded":"2023-12-04T05:40:01.501279+0000","last_fullsized":"2023-12-04T05:40:01.501279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:21:38.039584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683814+0000","last_change":"2023-12-04T05:39:59.731652+0000","last_active":"2023-12-04T05:40:00.683814+0000","last_peered":"2023-12-04T05:40:00.683814+0000","last_clean":"2023-12-04T05:40:00.683814+0000","last_became_active":"2023-12-04T05:39:59.730714+0000","last_became_peered":"2023-12-04T05:39:59.730714+0000","last_unstale":"2023-12-04T05:40:00.683814+0000","last_undegraded":"2023-12-04T05:40:00.683814+0000","last_fullsized":"2023-12-04T05:40:00.683814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:02:27.013221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.769202+0000","last_change":"2023-12-04T05:39:58.751530+0000","last_active":"2023-12-04T05:40:04.769202+0000","last_peered":"2023-12-04T05:40:04.769202+0000","last_clean":"2023-12-04T05:40:04.769202+0000","last_became_active":"2023-12-04T05:39:58.750857+0000","last_became_peered":"2023-12-04T05:39:58.750857+0000","last_unstale":"2023-12-04T05:40:04.769202+0000","last_undegraded":"2023-12-04T05:40:04.769202+0000","last_fullsized":"2023-12-04T05:40:04.769202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:43:36.369813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683195+0000","last_change":"2023-12-04T05:39:59.730535+0000","last_active":"2023-12-04T05:40:00.683195+0000","last_peered":"2023-12-04T05:40:00.683195+0000","last_clean":"2023-12-04T05:40:00.683195+0000","last_became_active":"2023-12-04T05:39:59.729717+0000","last_became_peered":"2023-12-04T05:39:59.729717+0000","last_unstale":"2023-12-04T05:40:00.683195+0000","last_undegraded":"2023-12-04T05:40:00.683195+0000","last_fullsized":"2023-12-04T05:40:00.683195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:31:46.120414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.770150+0000","last_change":"2023-12-04T05:39:58.752193+0000","last_active":"2023-12-04T05:40:04.770150+0000","last_peered":"2023-12-04T05:40:04.770150+0000","last_clean":"2023-12-04T05:40:04.770150+0000","last_became_active":"2023-12-04T05:39:58.751625+0000","last_became_peered":"2023-12-04T05:39:58.751625+0000","last_unstale":"2023-12-04T05:40:04.770150+0000","last_undegraded":"2023-12-04T05:40:04.770150+0000","last_fullsized":"2023-12-04T05:40:04.770150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:05:05.734494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501167+0000","last_change":"2023-12-04T05:39:59.720197+0000","last_active":"2023-12-04T05:40:01.501167+0000","last_peered":"2023-12-04T05:40:01.501167+0000","last_clean":"2023-12-04T05:40:01.501167+0000","last_became_active":"2023-12-04T05:39:59.718736+0000","last_became_peered":"2023-12-04T05:39:59.718736+0000","last_unstale":"2023-12-04T05:40:01.501167+0000","last_undegraded":"2023-12-04T05:40:01.501167+0000","last_fullsized":"2023-12-04T05:40:01.501167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:41:13.184198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.529254+0000","last_change":"2023-12-04T05:39:58.726906+0000","last_active":"2023-12-04T05:40:01.529254+0000","last_peered":"2023-12-04T05:40:01.529254+0000","last_clean":"2023-12-04T05:40:01.529254+0000","last_became_active":"2023-12-04T05:39:58.726415+0000","last_became_peered":"2023-12-04T05:39:58.726415+0000","last_unstale":"2023-12-04T05:40:01.529254+0000","last_undegraded":"2023-12-04T05:40:01.529254+0000","last_fullsized":"2023-12-04T05:40:01.529254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:58:53.715291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503085+0000","last_change":"2023-12-04T05:39:59.711861+0000","last_active":"2023-12-04T05:40:01.503085+0000","last_peered":"2023-12-04T05:40:01.503085+0000","last_clean":"2023-12-04T05:40:01.503085+0000","last_became_active":"2023-12-04T05:39:59.710933+0000","last_became_peered":"2023-12-04T05:39:59.710933+0000","last_unstale":"2023-12-04T05:40:01.503085+0000","last_undegraded":"2023-12-04T05:40:01.503085+0000","last_fullsized":"2023-12-04T05:40:01.503085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:40:46.361768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.771740+0000","last_change":"2023-12-04T05:39:58.740704+0000","last_active":"2023-12-04T05:40:04.771740+0000","last_peered":"2023-12-04T05:40:04.771740+0000","last_clean":"2023-12-04T05:40:04.771740+0000","last_became_active":"2023-12-04T05:39:58.740440+0000","last_became_peered":"2023-12-04T05:39:58.740440+0000","last_unstale":"2023-12-04T05:40:04.771740+0000","last_undegraded":"2023-12-04T05:40:04.771740+0000","last_fullsized":"2023-12-04T05:40:04.771740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:20:18.197917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":815104,"data_stored":666531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.111}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0169999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":221184,"data_stored":75671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1859999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.119}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2789999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.667}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":229376,"data_stored":75707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7629999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.222}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1279999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0880000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":75741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0649999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1200000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":823296,"data_stored":666587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1619999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.22}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-04T05:40:08.761 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-04T05:40:08.761 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-04T05:40:08.761 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-04T05:40:08.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-04T05:40:08.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.907+0000 7f3577c09640 1 Processor -- start 2023-12-04T05:40:08.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.907+0000 7f3577c09640 1 -- start start 2023-12-04T05:40:08.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.911+0000 7f3577c09640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3570072110 0x7f3570158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:08.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.911+0000 7f3577c09640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f35700c56f0 0x7f3570140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:08.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.911+0000 7f3577c09640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3570158da0 con 0x7f35700724f0 2023-12-04T05:40:08.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.911+0000 7f3577c09640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35701583d0 con 0x7f3570072110 2023-12-04T05:40:08.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.911+0000 7f3577c09640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35700c6170 con 0x7f35700c56f0 2023-12-04T05:40:08.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.911+0000 7f35759a5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3570072110 0x7f3570158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:08.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.911+0000 7f35759a5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3570072110 0x7f3570158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40934/0 (socket says 172.21.15.107:40934) 2023-12-04T05:40:08.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.911+0000 7f35759a5640 1 -- 172.21.15.107:0/555475682 learned_addr learned my addr 172.21.15.107:0/555475682 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:08.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.911+0000 7f35761a6640 1 --2- 172.21.15.107:0/555475682 >> v2:172.21.15.195:3300/0 conn(0x7f35700c56f0 0x7f3570140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:08.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.911+0000 7f35759a5640 1 -- 172.21.15.107:0/555475682 >> v2:172.21.15.195:3300/0 conn(0x7f35700c56f0 msgr2=0x7f3570140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:08.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.911+0000 7f35759a5640 1 --2- 172.21.15.107:0/555475682 >> v2:172.21.15.195:3300/0 conn(0x7f35700c56f0 0x7f3570140830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.911+0000 7f35759a5640 1 -- 172.21.15.107:0/555475682 >> v1:172.21.15.195:6789/0 conn(0x7f35700724f0 legacy=0x7f35700c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:08.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.911+0000 7f35759a5640 1 -- 172.21.15.107:0/555475682 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3570157da0 con 0x7f3570072110 2023-12-04T05:40:08.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.911+0000 7f35759a5640 1 --2- 172.21.15.107:0/555475682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3570072110 0x7f3570158860 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f35700c9080 tx=0x7f356c01f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd0dd5593a56c56 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:08.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.911+0000 7f35749a3640 1 -- 172.21.15.107:0/555475682 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c0af400 con 0x7f3570072110 2023-12-04T05:40:08.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.911+0000 7f35749a3640 1 -- 172.21.15.107:0/555475682 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356c0ab050 con 0x7f3570072110 2023-12-04T05:40:08.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.911+0000 7f35749a3640 1 -- 172.21.15.107:0/555475682 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c0af780 con 0x7f3570072110 2023-12-04T05:40:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f3577c09640 1 -- 172.21.15.107:0/555475682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3570072110 msgr2=0x7f3570158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f3577c09640 1 --2- 172.21.15.107:0/555475682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3570072110 0x7f3570158860 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f35700c9080 tx=0x7f356c01f180 comp rx=0 tx=0).stop 2023-12-04T05:40:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f3577c09640 1 -- 172.21.15.107:0/555475682 shutdown_connections 2023-12-04T05:40:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f3577c09640 1 --2- 172.21.15.107:0/555475682 >> v2:172.21.15.195:3300/0 conn(0x7f35700c56f0 0x7f3570140830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f3577c09640 1 --2- 172.21.15.107:0/555475682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3570072110 0x7f3570158860 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f3577c09640 1 -- 172.21.15.107:0/555475682 >> 172.21.15.107:0/555475682 conn(0x7f35700c7ff0 msgr2=0x7f35700c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f3577c09640 1 -- 172.21.15.107:0/555475682 shutdown_connections 2023-12-04T05:40:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f3577c09640 1 -- 172.21.15.107:0/555475682 wait complete. 2023-12-04T05:40:08.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f3577c09640 1 Processor -- start 2023-12-04T05:40:08.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f3577c09640 1 -- start start 2023-12-04T05:40:08.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f3577c09640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3570072110 0x7f357015a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:08.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f3577c09640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f35700724f0 0x7f357014fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:08.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f3577c09640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f35700c56f0 0x7f35700ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:08.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f3577c09640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f357014f740 con 0x7f3570072110 2023-12-04T05:40:08.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f3577c09640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35701ee050 con 0x7f35700724f0 2023-12-04T05:40:08.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f3577c09640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35700c75b0 con 0x7f35700c56f0 2023-12-04T05:40:08.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f35759a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3570072110 0x7f357015a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:08.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f35759a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3570072110 0x7f357015a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40940/0 (socket says 172.21.15.107:40940) 2023-12-04T05:40:08.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f35759a5640 1 -- 172.21.15.107:0/2486582442 learned_addr learned my addr 172.21.15.107:0/2486582442 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:08.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f35751a4640 1 --2- 172.21.15.107:0/2486582442 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f35700724f0 0x7f357014fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:08.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f35761a6640 1 --2- 172.21.15.107:0/2486582442 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f35700c56f0 0x7f35700ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:08.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f35759a5640 1 -- 172.21.15.107:0/2486582442 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f35700c56f0 msgr2=0x7f35700ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f35759a5640 1 --2- 172.21.15.107:0/2486582442 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f35700c56f0 0x7f35700ca6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f35759a5640 1 -- 172.21.15.107:0/2486582442 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f35700724f0 msgr2=0x7f357014fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:08.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f35759a5640 1 --2- 172.21.15.107:0/2486582442 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f35700724f0 0x7f357014fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:08.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f35759a5640 1 -- 172.21.15.107:0/2486582442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c009b30 con 0x7f3570072110 2023-12-04T05:40:08.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f35751a4640 1 --2- 172.21.15.107:0/2486582442 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f35700724f0 0x7f357014fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:08.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f35759a5640 1 --2- 172.21.15.107:0/2486582442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3570072110 0x7f357015a720 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f356c01cdb0 tx=0x7f356c01bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:08.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f35667fc640 1 -- 172.21.15.107:0/2486582442 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c0afe60 con 0x7f3570072110 2023-12-04T05:40:08.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f35667fc640 1 -- 172.21.15.107:0/2486582442 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356c0b9410 con 0x7f3570072110 2023-12-04T05:40:08.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f3577c09640 1 -- 172.21.15.107:0/2486582442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35701ee310 con 0x7f3570072110 2023-12-04T05:40:08.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f35667fc640 1 -- 172.21.15.107:0/2486582442 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c0b9780 con 0x7f3570072110 2023-12-04T05:40:08.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.915+0000 7f3577c09640 1 -- 172.21.15.107:0/2486582442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f357014f080 con 0x7f3570072110 2023-12-04T05:40:08.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.919+0000 7f3577c09640 1 -- 172.21.15.107:0/2486582442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3570067870 con 0x7f3570072110 2023-12-04T05:40:08.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.919+0000 7f35667fc640 1 -- 172.21.15.107:0/2486582442 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f356c0b6070 con 0x7f3570072110 2023-12-04T05:40:08.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.919+0000 7f35667fc640 1 --2- 172.21.15.107:0/2486582442 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3554066900 0x7f3554068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:08.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.919+0000 7f35667fc640 1 -- 172.21.15.107:0/2486582442 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f356c122590 con 0x7f3570072110 2023-12-04T05:40:08.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.919+0000 7f35751a4640 1 --2- 172.21.15.107:0/2486582442 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3554066900 0x7f3554068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:08.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.919+0000 7f35751a4640 1 --2- 172.21.15.107:0/2486582442 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3554066900 0x7f3554068dc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f35701e7d10 tx=0x7f3560009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:08.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:08.923+0000 7f35667fc640 1 -- 172.21.15.107:0/2486582442 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f356c0ef860 con 0x7f3570072110 2023-12-04T05:40:09.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:09.095+0000 7f3577c09640 1 -- 172.21.15.107:0/2486582442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3570158da0 con 0x7f3570072110 2023-12-04T05:40:09.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:09.095+0000 7f35667fc640 1 -- 172.21.15.107:0/2486582442 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7f3570158da0 con 0x7f3570072110 2023-12-04T05:40:09.099 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:09.099 INFO:teuthology.orchestra.run.smithi107.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-12-04T05:40:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:09.099+0000 7f3577c09640 1 -- 172.21.15.107:0/2486582442 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3554066900 msgr2=0x7f3554068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:09.099+0000 7f3577c09640 1 --2- 172.21.15.107:0/2486582442 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3554066900 0x7f3554068dc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f35701e7d10 tx=0x7f3560009040 comp rx=0 tx=0).stop 2023-12-04T05:40:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:09.099+0000 7f3577c09640 1 -- 172.21.15.107:0/2486582442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3570072110 msgr2=0x7f357015a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:09.099+0000 7f3577c09640 1 --2- 172.21.15.107:0/2486582442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3570072110 0x7f357015a720 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f356c01cdb0 tx=0x7f356c01bf20 comp rx=0 tx=0).stop 2023-12-04T05:40:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:09.099+0000 7f3577c09640 1 -- 172.21.15.107:0/2486582442 shutdown_connections 2023-12-04T05:40:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:09.099+0000 7f3577c09640 1 --2- 172.21.15.107:0/2486582442 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3554066900 0x7f3554068dc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:09.099+0000 7f3577c09640 1 --2- 172.21.15.107:0/2486582442 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f35700c56f0 0x7f35700ca6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:09.099+0000 7f3577c09640 1 --2- 172.21.15.107:0/2486582442 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f35700724f0 0x7f357014fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:09.099+0000 7f3577c09640 1 --2- 172.21.15.107:0/2486582442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3570072110 0x7f357015a720 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:09.099+0000 7f3577c09640 1 -- 172.21.15.107:0/2486582442 >> 172.21.15.107:0/2486582442 conn(0x7f35700c7ff0 msgr2=0x7f3570156160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:09.099+0000 7f3577c09640 1 -- 172.21.15.107:0/2486582442 shutdown_connections 2023-12-04T05:40:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:09.099+0000 7f3577c09640 1 -- 172.21.15.107:0/2486582442 wait complete. 2023-12-04T05:40:12.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-04T05:40:12.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.247+0000 7f71cbe0e640 1 Processor -- start 2023-12-04T05:40:12.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.247+0000 7f71cbe0e640 1 -- start start 2023-12-04T05:40:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.247+0000 7f71cbe0e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f71c4072110 0x7f71c4158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.247+0000 7f71cbe0e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c4158da0 con 0x7f71c40c56f0 2023-12-04T05:40:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71cbe0e640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c41583d0 con 0x7f71c40724f0 2023-12-04T05:40:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71cbe0e640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c40c6170 con 0x7f71c4072110 2023-12-04T05:40:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71ca3ab640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f71c40c56f0 0x7f71c4140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43572/0 (socket says 172.21.15.107:43572) 2023-12-04T05:40:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71ca3ab640 1 -- 172.21.15.107:0/4246574779 learned_addr learned my addr 172.21.15.107:0/4246574779 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71c9baa640 1 --2- 172.21.15.107:0/4246574779 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f71c4072110 0x7f71c4158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71c8ba8640 1 -- 172.21.15.107:0/4246574779 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84865883 0 0) 0x7f71bc0014a0 con 0x7f71c40c56f0 2023-12-04T05:40:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71c8ba8640 1 -- 172.21.15.107:0/4246574779 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71b0003530 con 0x7f71c40c56f0 2023-12-04T05:40:12.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71c8ba8640 1 -- 172.21.15.107:0/4246574779 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3478170174 0 0) 0x7f71b0003530 con 0x7f71c40c56f0 2023-12-04T05:40:12.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71c8ba8640 1 -- 172.21.15.107:0/4246574779 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f71c4072110 msgr2=0x7f71c4158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71c8ba8640 1 --2- 172.21.15.107:0/4246574779 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f71c4072110 0x7f71c4158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71c8ba8640 1 -- 172.21.15.107:0/4246574779 >> v1:172.21.15.195:6789/0 conn(0x7f71c40724f0 legacy=0x7f71c40c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71c8ba8640 1 -- 172.21.15.107:0/4246574779 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c4157da0 con 0x7f71c40c56f0 2023-12-04T05:40:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71c8ba8640 1 -- 172.21.15.107:0/4246574779 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f71bc001b90 con 0x7f71c40c56f0 2023-12-04T05:40:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71c8ba8640 1 -- 172.21.15.107:0/4246574779 >> v1:172.21.15.107:6789/0 conn(0x7f71c40c56f0 legacy=0x7f71c4140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71c8ba8640 1 --2- 172.21.15.107:0/4246574779 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71b00039b0 0x7f71b0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71c8ba8640 1 --2- 172.21.15.107:0/4246574779 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f71c4072110 0x7f71b00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71c8ba8640 1 --2- 172.21.15.107:0/4246574779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71b0004d10 0x7f71b0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71c8ba8640 1 -- 172.21.15.107:0/4246574779 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c40c6170 con 0x7f71b0004d10 2023-12-04T05:40:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71c8ba8640 1 -- 172.21.15.107:0/4246574779 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b000ba40 con 0x7f71b00039b0 2023-12-04T05:40:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71c8ba8640 1 -- 172.21.15.107:0/4246574779 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b000bc00 con 0x7f71c4072110 2023-12-04T05:40:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71ca3ab640 1 --2- 172.21.15.107:0/4246574779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71b0004d10 0x7f71b0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71c93a9640 1 --2- 172.21.15.107:0/4246574779 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f71c4072110 0x7f71b00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:12.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71c9baa640 1 --2- 172.21.15.107:0/4246574779 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71b00039b0 0x7f71b0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:12.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71ca3ab640 1 -- 172.21.15.107:0/4246574779 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f71c4072110 msgr2=0x7f71b00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:12.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71ca3ab640 1 --2- 172.21.15.107:0/4246574779 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f71c4072110 0x7f71b00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:12.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71ca3ab640 1 -- 172.21.15.107:0/4246574779 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71b00039b0 msgr2=0x7f71b0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:12.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71ca3ab640 1 --2- 172.21.15.107:0/4246574779 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71b00039b0 0x7f71b0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:12.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71ca3ab640 1 -- 172.21.15.107:0/4246574779 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b0003530 con 0x7f71b0004d10 2023-12-04T05:40:12.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.251+0000 7f71c9baa640 1 --2- 172.21.15.107:0/4246574779 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71b00039b0 0x7f71b0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:40:12.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.255+0000 7f71ca3ab640 1 --2- 172.21.15.107:0/4246574779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71b0004d10 0x7f71b0005190 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f71bc00a800 tx=0x7f71bc01d300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c88ac89d7cc9f439 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:12.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.255+0000 7f71c8ba8640 1 -- 172.21.15.107:0/4246574779 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71bc0ad630 con 0x7f71b0004d10 2023-12-04T05:40:12.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.255+0000 7f71c8ba8640 1 -- 172.21.15.107:0/4246574779 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71bc0a8070 con 0x7f71b0004d10 2023-12-04T05:40:12.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.255+0000 7f71c8ba8640 1 -- 172.21.15.107:0/4246574779 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71bc0ad9f0 con 0x7f71b0004d10 2023-12-04T05:40:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.255+0000 7f71cbe0e640 1 -- 172.21.15.107:0/4246574779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71b0004d10 msgr2=0x7f71b0005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.255+0000 7f71cbe0e640 1 --2- 172.21.15.107:0/4246574779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71b0004d10 0x7f71b0005190 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f71bc00a800 tx=0x7f71bc01d300 comp rx=0 tx=0).stop 2023-12-04T05:40:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.255+0000 7f71ca3ab640 1 -- 172.21.15.107:0/4246574779 reap_dead start 2023-12-04T05:40:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.255+0000 7f71cbe0e640 1 -- 172.21.15.107:0/4246574779 shutdown_connections 2023-12-04T05:40:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.255+0000 7f71cbe0e640 1 -- 172.21.15.107:0/4246574779 >> 172.21.15.107:0/4246574779 conn(0x7f71c40c7ff0 msgr2=0x7f71c40c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.255+0000 7f71cbe0e640 1 -- 172.21.15.107:0/4246574779 shutdown_connections 2023-12-04T05:40:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.255+0000 7f71cbe0e640 1 -- 172.21.15.107:0/4246574779 wait complete. 2023-12-04T05:40:12.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71cbe0e640 1 Processor -- start 2023-12-04T05:40:12.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71cbe0e640 1 -- start start 2023-12-04T05:40:12.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71cbe0e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f71c41f0bd0 0x7f71c41f0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:12.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71cbe0e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71c4153c40 0x7f71c41590d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:12.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71cbe0e640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71c415a6e0 0x7f71c414cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:12.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71cbe0e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c4157140 con 0x7f71c4153c40 2023-12-04T05:40:12.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71cbe0e640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c415a3a0 con 0x7f71c415a6e0 2023-12-04T05:40:12.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71cbe0e640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c4159610 con 0x7f71c41f0bd0 2023-12-04T05:40:12.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71c93a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71c4153c40 0x7f71c41590d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:12.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71c93a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71c4153c40 0x7f71c41590d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40954/0 (socket says 172.21.15.107:40954) 2023-12-04T05:40:12.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71c93a9640 1 -- 172.21.15.107:0/2242688380 learned_addr learned my addr 172.21.15.107:0/2242688380 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:12.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71ca3ab640 1 --2- 172.21.15.107:0/2242688380 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71c415a6e0 0x7f71c414cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:12.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71c9baa640 1 --2- 172.21.15.107:0/2242688380 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f71c41f0bd0 0x7f71c41f0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:12.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71c93a9640 1 -- 172.21.15.107:0/2242688380 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f71c41f0bd0 msgr2=0x7f71c41f0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:12.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71c93a9640 1 --2- 172.21.15.107:0/2242688380 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f71c41f0bd0 0x7f71c41f0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:12.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71c93a9640 1 -- 172.21.15.107:0/2242688380 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71c415a6e0 msgr2=0x7f71c414cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:12.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71c93a9640 1 --2- 172.21.15.107:0/2242688380 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71c415a6e0 0x7f71c414cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:12.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71c93a9640 1 -- 172.21.15.107:0/2242688380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c41f1630 con 0x7f71c4153c40 2023-12-04T05:40:12.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71c93a9640 1 --2- 172.21.15.107:0/2242688380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71c4153c40 0x7f71c41590d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f71b400f820 tx=0x7f71b400fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:12.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71ba7fc640 1 -- 172.21.15.107:0/2242688380 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b4001350 con 0x7f71c4153c40 2023-12-04T05:40:12.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71ba7fc640 1 -- 172.21.15.107:0/2242688380 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b40050d0 con 0x7f71c4153c40 2023-12-04T05:40:12.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71cbe0e640 1 -- 172.21.15.107:0/2242688380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c415bd30 con 0x7f71c4153c40 2023-12-04T05:40:12.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71ba7fc640 1 -- 172.21.15.107:0/2242688380 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b4005460 con 0x7f71c4153c40 2023-12-04T05:40:12.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71cbe0e640 1 -- 172.21.15.107:0/2242688380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c415a030 con 0x7f71c4153c40 2023-12-04T05:40:12.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71cbe0e640 1 -- 172.21.15.107:0/2242688380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c4067870 con 0x7f71c4153c40 2023-12-04T05:40:12.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.259+0000 7f71ba7fc640 1 -- 172.21.15.107:0/2242688380 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f71b400d730 con 0x7f71c4153c40 2023-12-04T05:40:12.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.263+0000 7f71ba7fc640 1 --2- 172.21.15.107:0/2242688380 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f71a8066900 0x7f71a8068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:12.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.263+0000 7f71ba7fc640 1 -- 172.21.15.107:0/2242688380 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71b4084430 con 0x7f71c4153c40 2023-12-04T05:40:12.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.263+0000 7f71c9baa640 1 --2- 172.21.15.107:0/2242688380 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f71a8066900 0x7f71a8068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:12.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.263+0000 7f71c9baa640 1 --2- 172.21.15.107:0/2242688380 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f71a8066900 0x7f71a8068dc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f71c0009a30 tx=0x7f71c0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:12.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.267+0000 7f71ba7fc640 1 -- 172.21.15.107:0/2242688380 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f71b40515e0 con 0x7f71c4153c40 2023-12-04T05:40:12.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.455+0000 7f71cbe0e640 1 -- 172.21.15.107:0/2242688380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f71c4156880 con 0x7f71c4153c40 2023-12-04T05:40:12.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.455+0000 7f71ba7fc640 1 -- 172.21.15.107:0/2242688380 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7f71b4054c00 con 0x7f71c4153c40 2023-12-04T05:40:12.458 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:12.458 INFO:teuthology.orchestra.run.smithi107.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-12-04T05:40:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.455+0000 7f71cbe0e640 1 -- 172.21.15.107:0/2242688380 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f71a8066900 msgr2=0x7f71a8068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.455+0000 7f71cbe0e640 1 --2- 172.21.15.107:0/2242688380 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f71a8066900 0x7f71a8068dc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f71c0009a30 tx=0x7f71c0008040 comp rx=0 tx=0).stop 2023-12-04T05:40:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.455+0000 7f71cbe0e640 1 -- 172.21.15.107:0/2242688380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71c4153c40 msgr2=0x7f71c41590d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.455+0000 7f71cbe0e640 1 --2- 172.21.15.107:0/2242688380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71c4153c40 0x7f71c41590d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f71b400f820 tx=0x7f71b400fce0 comp rx=0 tx=0).stop 2023-12-04T05:40:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.455+0000 7f71cbe0e640 1 -- 172.21.15.107:0/2242688380 shutdown_connections 2023-12-04T05:40:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.455+0000 7f71cbe0e640 1 --2- 172.21.15.107:0/2242688380 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f71a8066900 0x7f71a8068dc0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.455+0000 7f71cbe0e640 1 --2- 172.21.15.107:0/2242688380 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71c415a6e0 0x7f71c414cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.455+0000 7f71cbe0e640 1 --2- 172.21.15.107:0/2242688380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71c4153c40 0x7f71c41590d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.455+0000 7f71cbe0e640 1 --2- 172.21.15.107:0/2242688380 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f71c41f0bd0 0x7f71c41f0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.455+0000 7f71cbe0e640 1 -- 172.21.15.107:0/2242688380 >> 172.21.15.107:0/2242688380 conn(0x7f71c40c7ff0 msgr2=0x7f71c406d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.455+0000 7f71cbe0e640 1 -- 172.21.15.107:0/2242688380 shutdown_connections 2023-12-04T05:40:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:12.455+0000 7f71cbe0e640 1 -- 172.21.15.107:0/2242688380 wait complete. 2023-12-04T05:40:15.260 INFO:tasks.ceph.mds.f.smithi195.stderr:2023-12-04T05:40:15.255+0000 7f5bdadb8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-04T05:40:15.260 INFO:tasks.ceph.mds.f.smithi195.stderr:2023-12-04T05:40:15.255+0000 7f5bdadb8640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-12-04T05:40:15.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-04T05:40:15.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.607+0000 7f40647b3640 1 Processor -- start 2023-12-04T05:40:15.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.607+0000 7f40647b3640 1 -- start start 2023-12-04T05:40:15.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f40647b3640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f405c071c00 0x7f405c0c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:15.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f40647b3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f405c070380 0x7f405c070780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:15.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f40647b3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f405c0700c0 con 0x7f405c151180 2023-12-04T05:40:15.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f40647b3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f405c156df0 con 0x7f405c070380 2023-12-04T05:40:15.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f40647b3640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f405c1570d0 con 0x7f405c071c00 2023-12-04T05:40:15.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f4062d50640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f405c070380 0x7f405c070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:15.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f406254f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f405c151180 0x7f405c0714f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43584/0 (socket says 172.21.15.107:43584) 2023-12-04T05:40:15.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f406254f640 1 -- 172.21.15.107:0/2117845566 learned_addr learned my addr 172.21.15.107:0/2117845566 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:15.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f4061d4e640 1 --2- 172.21.15.107:0/2117845566 >> v2:172.21.15.195:3300/0 conn(0x7f405c071c00 0x7f405c0c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:15.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f4062d50640 1 -- 172.21.15.107:0/2117845566 >> v2:172.21.15.195:3300/0 conn(0x7f405c071c00 msgr2=0x7f405c0c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:15.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f4062d50640 1 --2- 172.21.15.107:0/2117845566 >> v2:172.21.15.195:3300/0 conn(0x7f405c071c00 0x7f405c0c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:15.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f4062d50640 1 -- 172.21.15.107:0/2117845566 >> v1:172.21.15.107:6789/0 conn(0x7f405c151180 legacy=0x7f405c0714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:15.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f4062d50640 1 -- 172.21.15.107:0/2117845566 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f405c1573b0 con 0x7f405c070380 2023-12-04T05:40:15.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f4062d50640 1 --2- 172.21.15.107:0/2117845566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f405c070380 0x7f405c070780 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f405c06e5e0 tx=0x7f405401ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=670dda6260c2f847 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:15.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f406154d640 1 -- 172.21.15.107:0/2117845566 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40540a7ab0 con 0x7f405c070380 2023-12-04T05:40:15.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f406154d640 1 -- 172.21.15.107:0/2117845566 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40540a7c10 con 0x7f405c070380 2023-12-04T05:40:15.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f406154d640 1 -- 172.21.15.107:0/2117845566 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40540af440 con 0x7f405c070380 2023-12-04T05:40:15.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f40647b3640 1 -- 172.21.15.107:0/2117845566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f405c070380 msgr2=0x7f405c070780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:15.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f40647b3640 1 --2- 172.21.15.107:0/2117845566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f405c070380 0x7f405c070780 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f405c06e5e0 tx=0x7f405401ea40 comp rx=0 tx=0).stop 2023-12-04T05:40:15.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f40647b3640 1 -- 172.21.15.107:0/2117845566 shutdown_connections 2023-12-04T05:40:15.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f40647b3640 1 --2- 172.21.15.107:0/2117845566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f405c070380 0x7f405c070780 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:15.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f40647b3640 1 --2- 172.21.15.107:0/2117845566 >> v2:172.21.15.195:3300/0 conn(0x7f405c071c00 0x7f405c0c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:15.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f40647b3640 1 -- 172.21.15.107:0/2117845566 >> 172.21.15.107:0/2117845566 conn(0x7f405c0c7ff0 msgr2=0x7f405c0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:15.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f40647b3640 1 -- 172.21.15.107:0/2117845566 shutdown_connections 2023-12-04T05:40:15.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.611+0000 7f40647b3640 1 -- 172.21.15.107:0/2117845566 wait complete. 2023-12-04T05:40:15.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f40647b3640 1 Processor -- start 2023-12-04T05:40:15.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f40647b3640 1 -- start start 2023-12-04T05:40:15.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f40647b3640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f405c070380 0x7f405c1e1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:15.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f40647b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f405c071c00 0x7f405c156140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:15.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f40647b3640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f405c151180 0x7f405c156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:15.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f40647b3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c1e0cd0 con 0x7f405c071c00 2023-12-04T05:40:15.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f40647b3640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c156bc0 con 0x7f405c070380 2023-12-04T05:40:15.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f40647b3640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f405c1eb5d0 con 0x7f405c151180 2023-12-04T05:40:15.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f4061d4e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f405c071c00 0x7f405c156140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:15.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f4061d4e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f405c071c00 0x7f405c156140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40972/0 (socket says 172.21.15.107:40972) 2023-12-04T05:40:15.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f4061d4e640 1 -- 172.21.15.107:0/2720493434 learned_addr learned my addr 172.21.15.107:0/2720493434 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:15.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f406254f640 1 --2- 172.21.15.107:0/2720493434 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f405c070380 0x7f405c1e1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:15.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f4062d50640 1 --2- 172.21.15.107:0/2720493434 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f405c151180 0x7f405c156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:15.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f4061d4e640 1 -- 172.21.15.107:0/2720493434 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f405c151180 msgr2=0x7f405c156680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:15.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f4061d4e640 1 --2- 172.21.15.107:0/2720493434 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f405c151180 0x7f405c156680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:15.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f4061d4e640 1 -- 172.21.15.107:0/2720493434 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f405c070380 msgr2=0x7f405c1e1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:15.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f4061d4e640 1 --2- 172.21.15.107:0/2720493434 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f405c070380 0x7f405c1e1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:15.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f4061d4e640 1 -- 172.21.15.107:0/2720493434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4054009a70 con 0x7f405c071c00 2023-12-04T05:40:15.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f4061d4e640 1 --2- 172.21.15.107:0/2720493434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f405c071c00 0x7f405c156140 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f404c00d830 tx=0x7f404c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:15.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f4052ffd640 1 -- 172.21.15.107:0/2720493434 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c00b6c0 con 0x7f405c071c00 2023-12-04T05:40:15.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f4052ffd640 1 -- 172.21.15.107:0/2720493434 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f404c00b820 con 0x7f405c071c00 2023-12-04T05:40:15.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f4052ffd640 1 -- 172.21.15.107:0/2720493434 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c00bb90 con 0x7f405c071c00 2023-12-04T05:40:15.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f40647b3640 1 -- 172.21.15.107:0/2720493434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f405c1ebe30 con 0x7f405c071c00 2023-12-04T05:40:15.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f40647b3640 1 -- 172.21.15.107:0/2720493434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f405c14cbc0 con 0x7f405c071c00 2023-12-04T05:40:15.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.615+0000 7f40647b3640 1 -- 172.21.15.107:0/2720493434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f405c1ebfc0 con 0x7f405c071c00 2023-12-04T05:40:15.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.619+0000 7f4052ffd640 1 -- 172.21.15.107:0/2720493434 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f404c01e460 con 0x7f405c071c00 2023-12-04T05:40:15.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.619+0000 7f4052ffd640 1 --2- 172.21.15.107:0/2720493434 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f40400668d0 0x7f4040068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:15.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.619+0000 7f4052ffd640 1 -- 172.21.15.107:0/2720493434 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f404c084bd0 con 0x7f405c071c00 2023-12-04T05:40:15.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.619+0000 7f406254f640 1 --2- 172.21.15.107:0/2720493434 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f40400668d0 0x7f4040068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:15.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.619+0000 7f406254f640 1 --2- 172.21.15.107:0/2720493434 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f40400668d0 0x7f4040068d90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4058001290 tx=0x7f405800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:15.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.623+0000 7f4052ffd640 1 -- 172.21.15.107:0/2720493434 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f404c051e10 con 0x7f405c071c00 2023-12-04T05:40:15.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.811+0000 7f40647b3640 1 -- 172.21.15.107:0/2720493434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f405c067870 con 0x7f405c071c00 2023-12-04T05:40:15.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.811+0000 7f4052ffd640 1 -- 172.21.15.107:0/2720493434 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7f404c055430 con 0x7f405c071c00 2023-12-04T05:40:15.815 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:15.815 INFO:teuthology.orchestra.run.smithi107.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-12-04T05:40:15.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.815+0000 7f40647b3640 1 -- 172.21.15.107:0/2720493434 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f40400668d0 msgr2=0x7f4040068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:15.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.815+0000 7f40647b3640 1 --2- 172.21.15.107:0/2720493434 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f40400668d0 0x7f4040068d90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4058001290 tx=0x7f405800b040 comp rx=0 tx=0).stop 2023-12-04T05:40:15.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.815+0000 7f40647b3640 1 -- 172.21.15.107:0/2720493434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f405c071c00 msgr2=0x7f405c156140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:15.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.815+0000 7f40647b3640 1 --2- 172.21.15.107:0/2720493434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f405c071c00 0x7f405c156140 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f404c00d830 tx=0x7f404c00dcf0 comp rx=0 tx=0).stop 2023-12-04T05:40:15.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.815+0000 7f40647b3640 1 -- 172.21.15.107:0/2720493434 shutdown_connections 2023-12-04T05:40:15.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.815+0000 7f40647b3640 1 --2- 172.21.15.107:0/2720493434 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f40400668d0 0x7f4040068d90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:15.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.815+0000 7f40647b3640 1 --2- 172.21.15.107:0/2720493434 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f405c151180 0x7f405c156680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:15.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.815+0000 7f40647b3640 1 --2- 172.21.15.107:0/2720493434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f405c071c00 0x7f405c156140 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:15.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.815+0000 7f40647b3640 1 --2- 172.21.15.107:0/2720493434 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f405c070380 0x7f405c1e1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:15.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.815+0000 7f40647b3640 1 -- 172.21.15.107:0/2720493434 >> 172.21.15.107:0/2720493434 conn(0x7f405c0c7ff0 msgr2=0x7f405c072090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:15.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.815+0000 7f40647b3640 1 -- 172.21.15.107:0/2720493434 shutdown_connections 2023-12-04T05:40:15.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:15.815+0000 7f40647b3640 1 -- 172.21.15.107:0/2720493434 wait complete. 2023-12-04T05:40:18.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-04T05:40:18.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.963+0000 7f019c743640 1 Processor -- start 2023-12-04T05:40:18.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.963+0000 7f019c743640 1 -- start start 2023-12-04T05:40:18.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.967+0000 7f019c743640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0194154020 0x7f019415a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:18.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.967+0000 7f019c743640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f01940d01c0 0x7f0194071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:18.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.967+0000 7f019c743640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019415ad50 con 0x7f0194072220 2023-12-04T05:40:18.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.967+0000 7f019c743640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01940ca230 con 0x7f01940d01c0 2023-12-04T05:40:18.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.967+0000 7f019c743640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01940c6770 con 0x7f0194154020 2023-12-04T05:40:18.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.967+0000 7f0199cde640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f01940d01c0 0x7f0194071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:18.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.967+0000 7f0199cde640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f01940d01c0 0x7f0194071ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52990/0 (socket says 172.21.15.107:52990) 2023-12-04T05:40:18.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.967+0000 7f0199cde640 1 -- 172.21.15.107:0/4143418804 learned_addr learned my addr 172.21.15.107:0/4143418804 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:18.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.967+0000 7f019a4df640 1 --2- 172.21.15.107:0/4143418804 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0194154020 0x7f019415a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:18.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.967+0000 7f0199cde640 1 -- 172.21.15.107:0/4143418804 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0194154020 msgr2=0x7f019415a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:18.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.967+0000 7f0199cde640 1 --2- 172.21.15.107:0/4143418804 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0194154020 0x7f019415a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:18.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.967+0000 7f0199cde640 1 -- 172.21.15.107:0/4143418804 >> v1:172.21.15.195:6789/0 conn(0x7f0194072220 legacy=0x7f01940c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:18.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.967+0000 7f0199cde640 1 -- 172.21.15.107:0/4143418804 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019415d710 con 0x7f01940d01c0 2023-12-04T05:40:18.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.967+0000 7f0199cde640 1 --2- 172.21.15.107:0/4143418804 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01940d01c0 0x7f0194071ce0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0184009870 tx=0x7f018401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74a07df156d8bc2f server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:18.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.967+0000 7f01994dd640 1 -- 172.21.15.107:0/4143418804 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01840a8a30 con 0x7f01940d01c0 2023-12-04T05:40:18.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.967+0000 7f01994dd640 1 -- 172.21.15.107:0/4143418804 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01840a8b90 con 0x7f01940d01c0 2023-12-04T05:40:18.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.967+0000 7f01994dd640 1 -- 172.21.15.107:0/4143418804 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01840b0440 con 0x7f01940d01c0 2023-12-04T05:40:18.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.967+0000 7f019c743640 1 -- 172.21.15.107:0/4143418804 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01940d01c0 msgr2=0x7f0194071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:18.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.967+0000 7f019c743640 1 --2- 172.21.15.107:0/4143418804 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01940d01c0 0x7f0194071ce0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0184009870 tx=0x7f018401ed30 comp rx=0 tx=0).stop 2023-12-04T05:40:18.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019c743640 1 -- 172.21.15.107:0/4143418804 shutdown_connections 2023-12-04T05:40:18.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019c743640 1 --2- 172.21.15.107:0/4143418804 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01940d01c0 0x7f0194071ce0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:18.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019c743640 1 --2- 172.21.15.107:0/4143418804 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0194154020 0x7f019415a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:18.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019c743640 1 -- 172.21.15.107:0/4143418804 >> 172.21.15.107:0/4143418804 conn(0x7f019406c6c0 msgr2=0x7f01940cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:18.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019c743640 1 -- 172.21.15.107:0/4143418804 shutdown_connections 2023-12-04T05:40:18.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019c743640 1 -- 172.21.15.107:0/4143418804 wait complete. 2023-12-04T05:40:18.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019c743640 1 Processor -- start 2023-12-04T05:40:18.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019c743640 1 -- start start 2023-12-04T05:40:18.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019c743640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0194072220 0x7f01940c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:18.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019c743640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f01940d01c0 0x7f0194158c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:18.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019c743640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0194154020 0x7f0194149d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:18.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019c743640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01940cf080 con 0x7f0194154020 2023-12-04T05:40:18.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019c743640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01940cdea0 con 0x7f01940d01c0 2023-12-04T05:40:18.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019c743640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01940c8490 con 0x7f0194072220 2023-12-04T05:40:18.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019ace0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0194154020 0x7f0194149d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:18.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019ace0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0194154020 0x7f0194149d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52994/0 (socket says 172.21.15.107:52994) 2023-12-04T05:40:18.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019ace0640 1 -- 172.21.15.107:0/3259118474 learned_addr learned my addr 172.21.15.107:0/3259118474 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:18.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019a4df640 1 --2- 172.21.15.107:0/3259118474 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0194072220 0x7f01940c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:18.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019ace0640 1 -- 172.21.15.107:0/3259118474 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0194072220 msgr2=0x7f01940c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:18.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019ace0640 1 --2- 172.21.15.107:0/3259118474 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0194072220 0x7f01940c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:18.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019ace0640 1 -- 172.21.15.107:0/3259118474 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f01940d01c0 msgr2=0x7f0194158c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:18.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f0199cde640 1 --2- 172.21.15.107:0/3259118474 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f01940d01c0 0x7f0194158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:18.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019ace0640 1 --2- 172.21.15.107:0/3259118474 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f01940d01c0 0x7f0194158c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:18.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019ace0640 1 -- 172.21.15.107:0/3259118474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01840094d0 con 0x7f0194154020 2023-12-04T05:40:18.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019a4df640 1 --2- 172.21.15.107:0/3259118474 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0194072220 0x7f01940c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:18.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019ace0640 1 --2- 172.21.15.107:0/3259118474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0194154020 0x7f0194149d90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f018c00c770 tx=0x7f018c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:18.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f018affd640 1 -- 172.21.15.107:0/3259118474 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018c003b80 con 0x7f0194154020 2023-12-04T05:40:18.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f018affd640 1 -- 172.21.15.107:0/3259118474 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f018c003ce0 con 0x7f0194154020 2023-12-04T05:40:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019c743640 1 -- 172.21.15.107:0/3259118474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01941f0580 con 0x7f0194154020 2023-12-04T05:40:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f018affd640 1 -- 172.21.15.107:0/3259118474 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018c00aa90 con 0x7f0194154020 2023-12-04T05:40:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019c743640 1 -- 172.21.15.107:0/3259118474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01940cd570 con 0x7f0194154020 2023-12-04T05:40:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.971+0000 7f019c743640 1 -- 172.21.15.107:0/3259118474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0194067870 con 0x7f0194154020 2023-12-04T05:40:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.975+0000 7f018affd640 1 -- 172.21.15.107:0/3259118474 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f018c00abf0 con 0x7f0194154020 2023-12-04T05:40:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.975+0000 7f018affd640 1 --2- 172.21.15.107:0/3259118474 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0178066900 0x7f0178068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.975+0000 7f018affd640 1 -- 172.21.15.107:0/3259118474 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f018c084250 con 0x7f0194154020 2023-12-04T05:40:18.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.975+0000 7f019a4df640 1 --2- 172.21.15.107:0/3259118474 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0178066900 0x7f0178068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:18.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.975+0000 7f019a4df640 1 --2- 172.21.15.107:0/3259118474 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0178066900 0x7f0178068dc0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0190009a40 tx=0x7f0190008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:18.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:18.979+0000 7f018affd640 1 -- 172.21.15.107:0/3259118474 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f018c051490 con 0x7f0194154020 2023-12-04T05:40:19.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:19.159+0000 7f019c743640 1 -- 172.21.15.107:0/3259118474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f01940cdfe0 con 0x7f0194154020 2023-12-04T05:40:19.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:19.159+0000 7f018affd640 1 -- 172.21.15.107:0/3259118474 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7f018c054ab0 con 0x7f0194154020 2023-12-04T05:40:19.163 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:19.163 INFO:teuthology.orchestra.run.smithi107.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-12-04T05:40:19.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:19.163+0000 7f019c743640 1 -- 172.21.15.107:0/3259118474 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0178066900 msgr2=0x7f0178068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:19.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:19.163+0000 7f019c743640 1 --2- 172.21.15.107:0/3259118474 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0178066900 0x7f0178068dc0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0190009a40 tx=0x7f0190008040 comp rx=0 tx=0).stop 2023-12-04T05:40:19.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:19.163+0000 7f019c743640 1 -- 172.21.15.107:0/3259118474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0194154020 msgr2=0x7f0194149d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:19.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:19.163+0000 7f019c743640 1 --2- 172.21.15.107:0/3259118474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0194154020 0x7f0194149d90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f018c00c770 tx=0x7f018c00cc30 comp rx=0 tx=0).stop 2023-12-04T05:40:19.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:19.163+0000 7f019c743640 1 -- 172.21.15.107:0/3259118474 shutdown_connections 2023-12-04T05:40:19.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:19.163+0000 7f019c743640 1 --2- 172.21.15.107:0/3259118474 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0178066900 0x7f0178068dc0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:19.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:19.163+0000 7f019c743640 1 --2- 172.21.15.107:0/3259118474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0194154020 0x7f0194149d90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:19.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:19.163+0000 7f019c743640 1 --2- 172.21.15.107:0/3259118474 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f01940d01c0 0x7f0194158c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:19.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:19.163+0000 7f019c743640 1 --2- 172.21.15.107:0/3259118474 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0194072220 0x7f01940c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:19.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:19.163+0000 7f019c743640 1 -- 172.21.15.107:0/3259118474 >> 172.21.15.107:0/3259118474 conn(0x7f019406c6c0 msgr2=0x7f01940caac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:19.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:19.163+0000 7f019c743640 1 -- 172.21.15.107:0/3259118474 shutdown_connections 2023-12-04T05:40:19.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:19.163+0000 7f019c743640 1 -- 172.21.15.107:0/3259118474 wait complete. 2023-12-04T05:40:20.260 INFO:tasks.ceph.mds.f.smithi195.stderr:2023-12-04T05:40:20.255+0000 7f5bdadb8640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-12-04T05:40:22.182 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-04T05:40:22.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.307+0000 7f5715b79640 1 Processor -- start 2023-12-04T05:40:22.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.307+0000 7f5715b79640 1 -- start start 2023-12-04T05:40:22.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f5715b79640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5710072110 0x7f5710158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:22.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f5715b79640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f57100724f0 0x7f57100c7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:22.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f5715b79640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5710158da0 con 0x7f57100c56f0 2023-12-04T05:40:22.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f5715b79640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57101583d0 con 0x7f57100724f0 2023-12-04T05:40:22.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f5715b79640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57100c6170 con 0x7f5710072110 2023-12-04T05:40:22.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f57100c56f0 0x7f5710140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40252/0 (socket says 172.21.15.107:40252) 2023-12-04T05:40:22.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570ffff640 1 -- 172.21.15.107:0/3995379682 learned_addr learned my addr 172.21.15.107:0/3995379682 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:22.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570f7fe640 1 --2- 172.21.15.107:0/3995379682 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5710072110 0x7f5710158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:22.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570effd640 1 --2- 172.21.15.107:0/3995379682 >> v2:172.21.15.195:3300/0 conn(0x7f57100724f0 0x7f57100c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:22.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570e7fc640 1 -- 172.21.15.107:0/3995379682 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536267432 0 0) 0x7f57080014a0 con 0x7f57100c56f0 2023-12-04T05:40:22.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570e7fc640 1 -- 172.21.15.107:0/3995379682 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56f8003530 con 0x7f57100c56f0 2023-12-04T05:40:22.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570e7fc640 1 -- 172.21.15.107:0/3995379682 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1536881011 0 0) 0x7f56f8003530 con 0x7f57100c56f0 2023-12-04T05:40:22.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570e7fc640 1 -- 172.21.15.107:0/3995379682 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5710072110 msgr2=0x7f5710158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:22.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570e7fc640 1 --2- 172.21.15.107:0/3995379682 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5710072110 0x7f5710158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:22.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570e7fc640 1 -- 172.21.15.107:0/3995379682 >> v2:172.21.15.195:3300/0 conn(0x7f57100724f0 msgr2=0x7f57100c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:22.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570e7fc640 1 --2- 172.21.15.107:0/3995379682 >> v2:172.21.15.195:3300/0 conn(0x7f57100724f0 0x7f57100c7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:22.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570e7fc640 1 -- 172.21.15.107:0/3995379682 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5710157da0 con 0x7f57100c56f0 2023-12-04T05:40:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570e7fc640 1 -- 172.21.15.107:0/3995379682 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f5708001b90 con 0x7f57100c56f0 2023-12-04T05:40:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570e7fc640 1 -- 172.21.15.107:0/3995379682 >> v1:172.21.15.107:6789/0 conn(0x7f57100c56f0 legacy=0x7f5710140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570e7fc640 1 --2- 172.21.15.107:0/3995379682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56f8003980 0x7f56f8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570e7fc640 1 --2- 172.21.15.107:0/3995379682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f56f80047d0 0x7f56f8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570e7fc640 1 --2- 172.21.15.107:0/3995379682 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5710072110 0x7f56f8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570e7fc640 1 -- 172.21.15.107:0/3995379682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57101583d0 con 0x7f56f8003980 2023-12-04T05:40:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570e7fc640 1 -- 172.21.15.107:0/3995379682 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57100c6170 con 0x7f56f80047d0 2023-12-04T05:40:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570e7fc640 1 -- 172.21.15.107:0/3995379682 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f800b990 con 0x7f5710072110 2023-12-04T05:40:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570f7fe640 1 --2- 172.21.15.107:0/3995379682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56f8003980 0x7f56f8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570effd640 1 --2- 172.21.15.107:0/3995379682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f56f80047d0 0x7f56f8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570ffff640 1 --2- 172.21.15.107:0/3995379682 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5710072110 0x7f56f8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570f7fe640 1 -- 172.21.15.107:0/3995379682 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5710072110 msgr2=0x7f56f8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570f7fe640 1 --2- 172.21.15.107:0/3995379682 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5710072110 0x7f56f8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570f7fe640 1 -- 172.21.15.107:0/3995379682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f56f80047d0 msgr2=0x7f56f8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570f7fe640 1 --2- 172.21.15.107:0/3995379682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f56f80047d0 0x7f56f8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:22.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570f7fe640 1 -- 172.21.15.107:0/3995379682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f8003530 con 0x7f56f8003980 2023-12-04T05:40:22.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570ffff640 1 --2- 172.21.15.107:0/3995379682 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5710072110 0x7f56f8005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:22.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570effd640 1 --2- 172.21.15.107:0/3995379682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f56f80047d0 0x7f56f8004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570f7fe640 1 --2- 172.21.15.107:0/3995379682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56f8003980 0x7f56f8003da0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5700009ae0 tx=0x7f570001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=186f170768e5d962 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570e7fc640 1 -- 172.21.15.107:0/3995379682 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57000a6440 con 0x7f56f8003980 2023-12-04T05:40:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570e7fc640 1 -- 172.21.15.107:0/3995379682 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57000a65a0 con 0x7f56f8003980 2023-12-04T05:40:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.311+0000 7f570e7fc640 1 -- 172.21.15.107:0/3995379682 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57000a6950 con 0x7f56f8003980 2023-12-04T05:40:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f5715b79640 1 -- 172.21.15.107:0/3995379682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56f8003980 msgr2=0x7f56f8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f5715b79640 1 --2- 172.21.15.107:0/3995379682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56f8003980 0x7f56f8003da0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5700009ae0 tx=0x7f570001ca50 comp rx=0 tx=0).stop 2023-12-04T05:40:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f570ffff640 1 -- 172.21.15.107:0/3995379682 reap_dead start 2023-12-04T05:40:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f5715b79640 1 -- 172.21.15.107:0/3995379682 shutdown_connections 2023-12-04T05:40:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f5715b79640 1 -- 172.21.15.107:0/3995379682 >> 172.21.15.107:0/3995379682 conn(0x7f57100c7ff0 msgr2=0x7f57100c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f5715b79640 1 -- 172.21.15.107:0/3995379682 shutdown_connections 2023-12-04T05:40:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f5715b79640 1 -- 172.21.15.107:0/3995379682 wait complete. 2023-12-04T05:40:22.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f5715b79640 1 Processor -- start 2023-12-04T05:40:22.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f5715b79640 1 -- start start 2023-12-04T05:40:22.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f5715b79640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f571013bf20 0x7f57101f3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:22.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f5715b79640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f571013ef50 0x7f571013f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:22.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f5715b79640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57101e6910 0x7f57101e6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:22.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f5715b79640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57101e64f0 con 0x7f571013ef50 2023-12-04T05:40:22.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f5715b79640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57101e72d0 con 0x7f57101e6910 2023-12-04T05:40:22.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f5715b79640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f571013cdd0 con 0x7f571013bf20 2023-12-04T05:40:22.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f570effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f571013ef50 0x7f571013f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:22.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f570effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f571013ef50 0x7f571013f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53024/0 (socket says 172.21.15.107:53024) 2023-12-04T05:40:22.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f570effd640 1 -- 172.21.15.107:0/3526776815 learned_addr learned my addr 172.21.15.107:0/3526776815 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:22.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f570f7fe640 1 --2- 172.21.15.107:0/3526776815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f571013bf20 0x7f57101f3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:22.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f570ffff640 1 --2- 172.21.15.107:0/3526776815 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57101e6910 0x7f57101e6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:22.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f570effd640 1 -- 172.21.15.107:0/3526776815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f571013bf20 msgr2=0x7f57101f3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:22.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f570effd640 1 --2- 172.21.15.107:0/3526776815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f571013bf20 0x7f57101f3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:22.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f570effd640 1 -- 172.21.15.107:0/3526776815 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57101e6910 msgr2=0x7f57101e6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:22.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f570effd640 1 --2- 172.21.15.107:0/3526776815 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57101e6910 0x7f57101e6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:22.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f570effd640 1 -- 172.21.15.107:0/3526776815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57101f3cc0 con 0x7f571013ef50 2023-12-04T05:40:22.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f570effd640 1 --2- 172.21.15.107:0/3526776815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f571013ef50 0x7f571013f3d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5704010950 tx=0x7f5704010e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:22.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f56f7fff640 1 -- 172.21.15.107:0/3526776815 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570400ec20 con 0x7f571013ef50 2023-12-04T05:40:22.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f56f7fff640 1 -- 172.21.15.107:0/3526776815 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570400eda0 con 0x7f571013ef50 2023-12-04T05:40:22.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f56f7fff640 1 -- 172.21.15.107:0/3526776815 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5704004260 con 0x7f571013ef50 2023-12-04T05:40:22.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f5715b79640 1 -- 172.21.15.107:0/3526776815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571013d020 con 0x7f571013ef50 2023-12-04T05:40:22.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.315+0000 7f5715b79640 1 -- 172.21.15.107:0/3526776815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f571013d5e0 con 0x7f571013ef50 2023-12-04T05:40:22.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.319+0000 7f5715b79640 1 -- 172.21.15.107:0/3526776815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57101e6630 con 0x7f571013ef50 2023-12-04T05:40:22.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.319+0000 7f56f7fff640 1 -- 172.21.15.107:0/3526776815 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f57040043c0 con 0x7f571013ef50 2023-12-04T05:40:22.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.319+0000 7f56f7fff640 1 --2- 172.21.15.107:0/3526776815 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f56fc066900 0x7f56fc068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:22.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.319+0000 7f570f7fe640 1 --2- 172.21.15.107:0/3526776815 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f56fc066900 0x7f56fc068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:22.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.319+0000 7f56f7fff640 1 -- 172.21.15.107:0/3526776815 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f57040842f0 con 0x7f571013ef50 2023-12-04T05:40:22.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.319+0000 7f570f7fe640 1 --2- 172.21.15.107:0/3526776815 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f56fc066900 0x7f56fc068dc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f57000ad000 tx=0x7f57000b3040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:22.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.323+0000 7f56f7fff640 1 -- 172.21.15.107:0/3526776815 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f57101e6630 con 0x7f571013ef50 2023-12-04T05:40:22.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.511+0000 7f5715b79640 1 -- 172.21.15.107:0/3526776815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5710067870 con 0x7f571013ef50 2023-12-04T05:40:22.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.511+0000 7f56f7fff640 1 -- 172.21.15.107:0/3526776815 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7f5704051550 con 0x7f571013ef50 2023-12-04T05:40:22.515 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:22.515 INFO:teuthology.orchestra.run.smithi107.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-12-04T05:40:22.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.515+0000 7f5715b79640 1 -- 172.21.15.107:0/3526776815 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f56fc066900 msgr2=0x7f56fc068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:22.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.515+0000 7f5715b79640 1 --2- 172.21.15.107:0/3526776815 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f56fc066900 0x7f56fc068dc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f57000ad000 tx=0x7f57000b3040 comp rx=0 tx=0).stop 2023-12-04T05:40:22.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.515+0000 7f5715b79640 1 -- 172.21.15.107:0/3526776815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f571013ef50 msgr2=0x7f571013f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:22.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.515+0000 7f5715b79640 1 --2- 172.21.15.107:0/3526776815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f571013ef50 0x7f571013f3d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5704010950 tx=0x7f5704010e10 comp rx=0 tx=0).stop 2023-12-04T05:40:22.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.515+0000 7f5715b79640 1 -- 172.21.15.107:0/3526776815 shutdown_connections 2023-12-04T05:40:22.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.515+0000 7f5715b79640 1 --2- 172.21.15.107:0/3526776815 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f56fc066900 0x7f56fc068dc0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:22.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.515+0000 7f5715b79640 1 --2- 172.21.15.107:0/3526776815 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57101e6910 0x7f57101e6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:22.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.515+0000 7f5715b79640 1 --2- 172.21.15.107:0/3526776815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f571013ef50 0x7f571013f3d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:22.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.515+0000 7f5715b79640 1 --2- 172.21.15.107:0/3526776815 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f571013bf20 0x7f57101f3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:22.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.515+0000 7f5715b79640 1 -- 172.21.15.107:0/3526776815 >> 172.21.15.107:0/3526776815 conn(0x7f57100c7ff0 msgr2=0x7f57100c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:22.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.515+0000 7f5715b79640 1 -- 172.21.15.107:0/3526776815 shutdown_connections 2023-12-04T05:40:22.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:22.515+0000 7f5715b79640 1 -- 172.21.15.107:0/3526776815 wait complete. 2023-12-04T05:40:25.261 INFO:tasks.ceph.mds.f.smithi195.stderr:2023-12-04T05:40:25.255+0000 7f5bdadb8640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-12-04T05:40:25.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-04T05:40:25.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.659+0000 7fee3e0db640 1 Processor -- start 2023-12-04T05:40:25.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.659+0000 7fee3e0db640 1 -- start start 2023-12-04T05:40:25.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.659+0000 7fee3e0db640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fee38151180 0x7fee380c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.659+0000 7fee3e0db640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fee380714f0 0x7fee380718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.659+0000 7fee3e0db640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fee38070380 0x7fee38070790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.659+0000 7fee3e0db640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee380700c0 con 0x7fee38070380 2023-12-04T05:40:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.659+0000 7fee3e0db640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee38156df0 con 0x7fee380714f0 2023-12-04T05:40:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.659+0000 7fee3e0db640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee381570d0 con 0x7fee38151180 2023-12-04T05:40:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee37fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fee38070380 0x7fee38070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee37fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fee38070380 0x7fee38070790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53030/0 (socket says 172.21.15.107:53030) 2023-12-04T05:40:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee37fff640 1 -- 172.21.15.107:0/1150568135 learned_addr learned my addr 172.21.15.107:0/1150568135 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee377fe640 1 --2- 172.21.15.107:0/1150568135 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fee38151180 0x7fee380c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee36ffd640 1 --2- 172.21.15.107:0/1150568135 >> v2:172.21.15.195:3300/0 conn(0x7fee380714f0 0x7fee380718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee37fff640 1 -- 172.21.15.107:0/1150568135 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fee38151180 msgr2=0x7fee380c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee37fff640 1 --2- 172.21.15.107:0/1150568135 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fee38151180 0x7fee380c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee37fff640 1 -- 172.21.15.107:0/1150568135 >> v2:172.21.15.195:3300/0 conn(0x7fee380714f0 msgr2=0x7fee380718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee37fff640 1 --2- 172.21.15.107:0/1150568135 >> v2:172.21.15.195:3300/0 conn(0x7fee380714f0 0x7fee380718f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee37fff640 1 -- 172.21.15.107:0/1150568135 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee381573b0 con 0x7fee38070380 2023-12-04T05:40:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee36ffd640 1 --2- 172.21.15.107:0/1150568135 >> v2:172.21.15.195:3300/0 conn(0x7fee380714f0 0x7fee380718f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee37fff640 1 --2- 172.21.15.107:0/1150568135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee38070380 0x7fee38070790 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fee3806ba50 tx=0x7fee3001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57015285d39dcc94 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee367fc640 1 -- 172.21.15.107:0/1150568135 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee300a89b0 con 0x7fee38070380 2023-12-04T05:40:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee367fc640 1 -- 172.21.15.107:0/1150568135 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee300a8b10 con 0x7fee38070380 2023-12-04T05:40:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee367fc640 1 -- 172.21.15.107:0/1150568135 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee300a8ea0 con 0x7fee38070380 2023-12-04T05:40:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee3e0db640 1 -- 172.21.15.107:0/1150568135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee38070380 msgr2=0x7fee38070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee3e0db640 1 --2- 172.21.15.107:0/1150568135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee38070380 0x7fee38070790 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fee3806ba50 tx=0x7fee3001ecf0 comp rx=0 tx=0).stop 2023-12-04T05:40:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee3e0db640 1 -- 172.21.15.107:0/1150568135 shutdown_connections 2023-12-04T05:40:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee3e0db640 1 --2- 172.21.15.107:0/1150568135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee38070380 0x7fee38070790 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee3e0db640 1 --2- 172.21.15.107:0/1150568135 >> v2:172.21.15.195:3300/0 conn(0x7fee380714f0 0x7fee380718f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee3e0db640 1 --2- 172.21.15.107:0/1150568135 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fee38151180 0x7fee380c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee3e0db640 1 -- 172.21.15.107:0/1150568135 >> 172.21.15.107:0/1150568135 conn(0x7fee380c7ff0 msgr2=0x7fee380c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee3e0db640 1 -- 172.21.15.107:0/1150568135 shutdown_connections 2023-12-04T05:40:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.663+0000 7fee3e0db640 1 -- 172.21.15.107:0/1150568135 wait complete. 2023-12-04T05:40:25.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee3e0db640 1 Processor -- start 2023-12-04T05:40:25.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee3e0db640 1 -- start start 2023-12-04T05:40:25.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee3e0db640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fee38070380 0x7fee3815a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:25.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee3e0db640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee380714f0 0x7fee380cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:25.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee3e0db640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee38151180 0x7fee38155b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:25.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee3e0db640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee38159c30 con 0x7fee38151180 2023-12-04T05:40:25.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee3e0db640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee38157e30 con 0x7fee380714f0 2023-12-04T05:40:25.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee3e0db640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee381ef820 con 0x7fee38070380 2023-12-04T05:40:25.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee37fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee38151180 0x7fee38155b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:25.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee37fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee38151180 0x7fee38155b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53044/0 (socket says 172.21.15.107:53044) 2023-12-04T05:40:25.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee37fff640 1 -- 172.21.15.107:0/3384406050 learned_addr learned my addr 172.21.15.107:0/3384406050 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:25.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee377fe640 1 --2- 172.21.15.107:0/3384406050 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fee38070380 0x7fee3815a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:25.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee36ffd640 1 --2- 172.21.15.107:0/3384406050 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee380714f0 0x7fee380cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:25.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee37fff640 1 -- 172.21.15.107:0/3384406050 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fee38070380 msgr2=0x7fee3815a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:25.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee37fff640 1 --2- 172.21.15.107:0/3384406050 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fee38070380 0x7fee3815a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:25.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee37fff640 1 -- 172.21.15.107:0/3384406050 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee380714f0 msgr2=0x7fee380cacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:25.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee37fff640 1 --2- 172.21.15.107:0/3384406050 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee380714f0 0x7fee380cacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:25.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee37fff640 1 -- 172.21.15.107:0/3384406050 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee300094d0 con 0x7fee38151180 2023-12-04T05:40:25.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee36ffd640 1 --2- 172.21.15.107:0/3384406050 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee380714f0 0x7fee380cacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:25.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee37fff640 1 --2- 172.21.15.107:0/3384406050 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee38151180 0x7fee38155b10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fee3001f590 tx=0x7fee3001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:25.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee2ffff640 1 -- 172.21.15.107:0/3384406050 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee300ba600 con 0x7fee38151180 2023-12-04T05:40:25.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee2ffff640 1 -- 172.21.15.107:0/3384406050 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee300ba760 con 0x7fee38151180 2023-12-04T05:40:25.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee3e0db640 1 -- 172.21.15.107:0/3384406050 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee38154d50 con 0x7fee38151180 2023-12-04T05:40:25.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee2ffff640 1 -- 172.21.15.107:0/3384406050 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee300bab10 con 0x7fee38151180 2023-12-04T05:40:25.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee3e0db640 1 -- 172.21.15.107:0/3384406050 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee38155310 con 0x7fee38151180 2023-12-04T05:40:25.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee3e0db640 1 -- 172.21.15.107:0/3384406050 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee38067870 con 0x7fee38151180 2023-12-04T05:40:25.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.667+0000 7fee2ffff640 1 -- 172.21.15.107:0/3384406050 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fee300bac70 con 0x7fee38151180 2023-12-04T05:40:25.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.671+0000 7fee2ffff640 1 --2- 172.21.15.107:0/3384406050 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fee18066900 0x7fee18068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:25.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.671+0000 7fee377fe640 1 --2- 172.21.15.107:0/3384406050 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fee18066900 0x7fee18068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:25.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.671+0000 7fee2ffff640 1 -- 172.21.15.107:0/3384406050 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee30122a40 con 0x7fee38151180 2023-12-04T05:40:25.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.671+0000 7fee377fe640 1 --2- 172.21.15.107:0/3384406050 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fee18066900 0x7fee18068dc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fee28003fd0 tx=0x7fee280039a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:25.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.675+0000 7fee2ffff640 1 -- 172.21.15.107:0/3384406050 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fee30124e60 con 0x7fee38151180 2023-12-04T05:40:25.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.859+0000 7fee3e0db640 1 -- 172.21.15.107:0/3384406050 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fee380c8ba0 con 0x7fee38151180 2023-12-04T05:40:25.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.859+0000 7fee2ffff640 1 -- 172.21.15.107:0/3384406050 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fee300b5080 con 0x7fee38151180 2023-12-04T05:40:25.867 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:25.867 INFO:teuthology.orchestra.run.smithi107.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-04T05:40:25.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.863+0000 7fee3e0db640 1 -- 172.21.15.107:0/3384406050 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fee18066900 msgr2=0x7fee18068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:25.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.863+0000 7fee3e0db640 1 --2- 172.21.15.107:0/3384406050 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fee18066900 0x7fee18068dc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fee28003fd0 tx=0x7fee280039a0 comp rx=0 tx=0).stop 2023-12-04T05:40:25.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.863+0000 7fee3e0db640 1 -- 172.21.15.107:0/3384406050 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee38151180 msgr2=0x7fee38155b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:25.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.863+0000 7fee3e0db640 1 --2- 172.21.15.107:0/3384406050 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee38151180 0x7fee38155b10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fee3001f590 tx=0x7fee3001ba90 comp rx=0 tx=0).stop 2023-12-04T05:40:25.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.863+0000 7fee3e0db640 1 -- 172.21.15.107:0/3384406050 shutdown_connections 2023-12-04T05:40:25.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.863+0000 7fee3e0db640 1 --2- 172.21.15.107:0/3384406050 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fee18066900 0x7fee18068dc0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:25.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.863+0000 7fee3e0db640 1 --2- 172.21.15.107:0/3384406050 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee38151180 0x7fee38155b10 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:25.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.863+0000 7fee3e0db640 1 --2- 172.21.15.107:0/3384406050 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee380714f0 0x7fee380cacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:25.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.863+0000 7fee3e0db640 1 --2- 172.21.15.107:0/3384406050 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fee38070380 0x7fee3815a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:25.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.863+0000 7fee3e0db640 1 -- 172.21.15.107:0/3384406050 >> 172.21.15.107:0/3384406050 conn(0x7fee380c7ff0 msgr2=0x7fee3814f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:25.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.863+0000 7fee3e0db640 1 -- 172.21.15.107:0/3384406050 shutdown_connections 2023-12-04T05:40:25.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.863+0000 7fee3e0db640 1 -- 172.21.15.107:0/3384406050 wait complete. 2023-12-04T05:40:25.882 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-04T05:40:25.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T05:40:26.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.995+0000 7f21aec28640 1 Processor -- start 2023-12-04T05:40:26.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.999+0000 7f21aec28640 1 -- start start 2023-12-04T05:40:26.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.999+0000 7f21aec28640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f21a8154290 0x7f21a806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:26.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.999+0000 7f21aec28640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f21a80c6b70 0x7f21a815a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:26.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.999+0000 7f21aec28640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a806e320 con 0x7f21a80c8c80 2023-12-04T05:40:26.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.999+0000 7f21aec28640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a8071c10 con 0x7f21a8154290 2023-12-04T05:40:26.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.999+0000 7f21aec28640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a80c66e0 con 0x7f21a80c6b70 2023-12-04T05:40:26.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.999+0000 7f21ac9c4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f21a8154290 0x7f21a806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:26.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.999+0000 7f21ac9c4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f21a8154290 0x7f21a806fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53048/0 (socket says 172.21.15.107:53048) 2023-12-04T05:40:26.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.999+0000 7f21ac9c4640 1 -- 172.21.15.107:0/4183298117 learned_addr learned my addr 172.21.15.107:0/4183298117 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:26.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.999+0000 7f21ad1c5640 1 --2- 172.21.15.107:0/4183298117 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f21a80c6b70 0x7f21a815a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:26.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.999+0000 7f21ac9c4640 1 -- 172.21.15.107:0/4183298117 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f21a80c6b70 msgr2=0x7f21a815a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:26.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.999+0000 7f21ac9c4640 1 --2- 172.21.15.107:0/4183298117 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f21a80c6b70 0x7f21a815a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.999+0000 7f21ac9c4640 1 -- 172.21.15.107:0/4183298117 >> v1:172.21.15.195:6789/0 conn(0x7f21a80c8c80 legacy=0x7f21a815a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:26.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.999+0000 7f21ac9c4640 1 -- 172.21.15.107:0/4183298117 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a80c7000 con 0x7f21a8154290 2023-12-04T05:40:26.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.999+0000 7f21ac9c4640 1 --2- 172.21.15.107:0/4183298117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a8154290 0x7f21a806fca0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2198009870 tx=0x7f219801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=893cad993861d18a server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:26.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.999+0000 7f21a77fe640 1 -- 172.21.15.107:0/4183298117 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21980ab600 con 0x7f21a8154290 2023-12-04T05:40:26.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.999+0000 7f21a77fe640 1 -- 172.21.15.107:0/4183298117 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21980a7050 con 0x7f21a8154290 2023-12-04T05:40:26.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:25.999+0000 7f21a77fe640 1 -- 172.21.15.107:0/4183298117 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21980ab980 con 0x7f21a8154290 2023-12-04T05:40:26.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21aec28640 1 -- 172.21.15.107:0/4183298117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a8154290 msgr2=0x7f21a806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:26.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21aec28640 1 --2- 172.21.15.107:0/4183298117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a8154290 0x7f21a806fca0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2198009870 tx=0x7f219801c3e0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21aec28640 1 -- 172.21.15.107:0/4183298117 shutdown_connections 2023-12-04T05:40:26.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21aec28640 1 --2- 172.21.15.107:0/4183298117 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f21a80c6b70 0x7f21a815a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21aec28640 1 --2- 172.21.15.107:0/4183298117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a8154290 0x7f21a806fca0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21aec28640 1 -- 172.21.15.107:0/4183298117 >> 172.21.15.107:0/4183298117 conn(0x7f21a80c7ff0 msgr2=0x7f21a80c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:26.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21aec28640 1 -- 172.21.15.107:0/4183298117 shutdown_connections 2023-12-04T05:40:26.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21aec28640 1 -- 172.21.15.107:0/4183298117 wait complete. 2023-12-04T05:40:26.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21aec28640 1 Processor -- start 2023-12-04T05:40:26.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21aec28640 1 -- start start 2023-12-04T05:40:26.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21aec28640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21a80c6b70 0x7f21a81f31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:26.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21aec28640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f21a80c8c80 0x7f21a81f36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:26.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21aec28640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a8154290 0x7f21a80c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:26.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21aec28640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a80c2c70 con 0x7f21a8154290 2023-12-04T05:40:26.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21aec28640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a81515e0 con 0x7f21a80c6b70 2023-12-04T05:40:26.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21aec28640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a81f6fd0 con 0x7f21a80c8c80 2023-12-04T05:40:26.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21ad1c5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a8154290 0x7f21a80c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:26.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21ad1c5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a8154290 0x7f21a80c35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53064/0 (socket says 172.21.15.107:53064) 2023-12-04T05:40:26.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21ad1c5640 1 -- 172.21.15.107:0/159151931 learned_addr learned my addr 172.21.15.107:0/159151931 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:26.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21ad1c5640 1 -- 172.21.15.107:0/159151931 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f21a80c8c80 msgr2=0x7f21a81f36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:26.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21ad1c5640 1 --2- 172.21.15.107:0/159151931 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f21a80c8c80 0x7f21a81f36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21ad1c5640 1 -- 172.21.15.107:0/159151931 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21a80c6b70 msgr2=0x7f21a81f31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:26.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21ad1c5640 1 --2- 172.21.15.107:0/159151931 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21a80c6b70 0x7f21a81f31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21ad1c5640 1 -- 172.21.15.107:0/159151931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21980094d0 con 0x7f21a8154290 2023-12-04T05:40:26.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21ad1c5640 1 --2- 172.21.15.107:0/159151931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a8154290 0x7f21a80c35c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f21a0007e40 tx=0x7f21a000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:26.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21a57fa640 1 -- 172.21.15.107:0/159151931 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a0014620 con 0x7f21a8154290 2023-12-04T05:40:26.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21a57fa640 1 -- 172.21.15.107:0/159151931 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a000ee90 con 0x7f21a8154290 2023-12-04T05:40:26.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21aec28640 1 -- 172.21.15.107:0/159151931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a81f72b0 con 0x7f21a8154290 2023-12-04T05:40:26.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21a57fa640 1 -- 172.21.15.107:0/159151931 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a00149e0 con 0x7f21a8154290 2023-12-04T05:40:26.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.003+0000 7f21aec28640 1 -- 172.21.15.107:0/159151931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a8151230 con 0x7f21a8154290 2023-12-04T05:40:26.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.007+0000 7f21aec28640 1 -- 172.21.15.107:0/159151931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a81f74f0 con 0x7f21a8154290 2023-12-04T05:40:26.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.007+0000 7f21a57fa640 1 -- 172.21.15.107:0/159151931 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f21a0014b40 con 0x7f21a8154290 2023-12-04T05:40:26.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.007+0000 7f21a57fa640 1 --2- 172.21.15.107:0/159151931 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f21940668d0 0x7f2194068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:26.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.007+0000 7f21a57fa640 1 -- 172.21.15.107:0/159151931 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f21a0084dd0 con 0x7f21a8154290 2023-12-04T05:40:26.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.007+0000 7f21ac9c4640 1 --2- 172.21.15.107:0/159151931 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f21940668d0 0x7f2194068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:26.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.007+0000 7f21ac9c4640 1 --2- 172.21.15.107:0/159151931 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f21940668d0 0x7f2194068d90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f21980ad000 tx=0x7f2198008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:26.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.011+0000 7f21a57fa640 1 -- 172.21.15.107:0/159151931 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f21a0052010 con 0x7f21a8154290 2023-12-04T05:40:26.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.167+0000 7f21aec28640 1 -- 172.21.15.107:0/159151931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21a8072dc0 con 0x7f21a8154290 2023-12-04T05:40:26.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.167+0000 7f21a57fa640 1 -- 172.21.15.107:0/159151931 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f21a0055630 con 0x7f21a8154290 2023-12-04T05:40:26.170 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:26.171 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":21,"fsid":"603e856b-31ac-475a-8f35-21ab7faadd48","created":"2023-12-04T05:39:34.921690+0000","modified":"2023-12-04T05:40:00.671650+0000","last_up_change":"2023-12-04T05:39:46.235790+0000","last_in_change":"2023-12-04T05:39:37.587639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T05:39:47.567702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T05:39:57.644286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T05:39:58.029897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ab7111c-0ae8-4d41-b6db-e3743614322c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6817","nonce":768714279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6821","nonce":768714279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6829","nonce":768714279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6825","nonce":768714279}]},"public_addr":"172.21.15.107:6817/768714279","cluster_addr":"172.21.15.107:6821/768714279","heartbeat_back_addr":"172.21.15.107:6829/768714279","heartbeat_front_addr":"172.21.15.107:6825/768714279","state":["exists","up"]},{"osd":1,"uuid":"baef7a1f-8026-4fab-9e70-48465271034e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6801","nonce":2496355981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6803","nonce":2496355981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6807","nonce":2496355981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6805","nonce":2496355981}]},"public_addr":"172.21.15.107:6801/2496355981","cluster_addr":"172.21.15.107:6803/2496355981","heartbeat_back_addr":"172.21.15.107:6807/2496355981","heartbeat_front_addr":"172.21.15.107:6805/2496355981","state":["exists","up"]},{"osd":2,"uuid":"90e8cf24-7b54-484a-9d85-b44a9d3dfeab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6809","nonce":3097298224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6811","nonce":3097298224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6815","nonce":3097298224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6813","nonce":3097298224}]},"public_addr":"172.21.15.107:6809/3097298224","cluster_addr":"172.21.15.107:6811/3097298224","heartbeat_back_addr":"172.21.15.107:6815/3097298224","heartbeat_front_addr":"172.21.15.107:6813/3097298224","state":["exists","up"]},{"osd":3,"uuid":"54658731-2b9a-4d5e-ab13-7c1be7d34e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6820","nonce":3773076671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6824","nonce":3773076671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6831","nonce":3773076671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6828","nonce":3773076671}]},"public_addr":"172.21.15.107:6820/3773076671","cluster_addr":"172.21.15.107:6824/3773076671","heartbeat_back_addr":"172.21.15.107:6831/3773076671","heartbeat_front_addr":"172.21.15.107:6828/3773076671","state":["exists","up"]},{"osd":4,"uuid":"9a9dea13-8afe-485d-aa43-45eda691179b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6801","nonce":2747823542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6803","nonce":2747823542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6807","nonce":2747823542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6805","nonce":2747823542}]},"public_addr":"172.21.15.195:6801/2747823542","cluster_addr":"172.21.15.195:6803/2747823542","heartbeat_back_addr":"172.21.15.195:6807/2747823542","heartbeat_front_addr":"172.21.15.195:6805/2747823542","state":["exists","up"]},{"osd":5,"uuid":"db840d83-5a7d-468a-8aa2-1997945847c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6825","nonce":1484264540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6827","nonce":1484264540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6831","nonce":1484264540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6829","nonce":1484264540}]},"public_addr":"172.21.15.195:6825/1484264540","cluster_addr":"172.21.15.195:6827/1484264540","heartbeat_back_addr":"172.21.15.195:6831/1484264540","heartbeat_front_addr":"172.21.15.195:6829/1484264540","state":["exists","up"]},{"osd":6,"uuid":"38b5f165-bf91-467e-a615-4b4784337689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6811","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6813","nonce":2638471412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6817","nonce":2638471412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6823","nonce":2638471412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6821","nonce":2638471412}]},"public_addr":"172.21.15.195:6813/2638471412","cluster_addr":"172.21.15.195:6817/2638471412","heartbeat_back_addr":"172.21.15.195:6823/2638471412","heartbeat_front_addr":"172.21.15.195:6821/2638471412","state":["exists","up"]},{"osd":7,"uuid":"e5b3c596-494b-4df3-9437-bc2d52168ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6809","nonce":768189614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6812","nonce":768189614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6820","nonce":768189614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6816","nonce":768189614}]},"public_addr":"172.21.15.195:6809/768189614","cluster_addr":"172.21.15.195:6812/768189614","heartbeat_back_addr":"172.21.15.195:6820/768189614","heartbeat_front_addr":"172.21.15.195:6816/768189614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.209077+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.086601+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.221568+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.238524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.059477+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.148422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.079867+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.064916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T05:40:26.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.171+0000 7f21aec28640 1 -- 172.21.15.107:0/159151931 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f21940668d0 msgr2=0x7f2194068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:26.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.171+0000 7f21aec28640 1 --2- 172.21.15.107:0/159151931 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f21940668d0 0x7f2194068d90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f21980ad000 tx=0x7f2198008040 comp rx=0 tx=0).stop 2023-12-04T05:40:26.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.171+0000 7f21aec28640 1 -- 172.21.15.107:0/159151931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a8154290 msgr2=0x7f21a80c35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:26.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.171+0000 7f21aec28640 1 --2- 172.21.15.107:0/159151931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a8154290 0x7f21a80c35c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f21a0007e40 tx=0x7f21a000e4b0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.171+0000 7f21aec28640 1 -- 172.21.15.107:0/159151931 shutdown_connections 2023-12-04T05:40:26.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.171+0000 7f21aec28640 1 --2- 172.21.15.107:0/159151931 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f21940668d0 0x7f2194068d90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.171+0000 7f21aec28640 1 --2- 172.21.15.107:0/159151931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a8154290 0x7f21a80c35c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.171+0000 7f21aec28640 1 --2- 172.21.15.107:0/159151931 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f21a80c8c80 0x7f21a81f36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.171+0000 7f21aec28640 1 --2- 172.21.15.107:0/159151931 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21a80c6b70 0x7f21a81f31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.171+0000 7f21aec28640 1 -- 172.21.15.107:0/159151931 >> 172.21.15.107:0/159151931 conn(0x7f21a80c7ff0 msgr2=0x7f21a81474d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:26.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.171+0000 7f21aec28640 1 -- 172.21.15.107:0/159151931 shutdown_connections 2023-12-04T05:40:26.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.171+0000 7f21aec28640 1 -- 172.21.15.107:0/159151931 wait complete. 2023-12-04T05:40:26.189 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-04T05:39:47.567702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-04T05:39:57.644286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-04T05:39:58.029897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-04T05:40:26.190 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-04T05:40:26.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.311+0000 7f156820b640 1 Processor -- start 2023-12-04T05:40:26.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.311+0000 7f156820b640 1 -- start start 2023-12-04T05:40:26.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.315+0000 7f156820b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1560072110 0x7f1560158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:26.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.315+0000 7f156820b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1560158da0 con 0x7f15600c56f0 2023-12-04T05:40:26.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.315+0000 7f156820b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15601583d0 con 0x7f15600724f0 2023-12-04T05:40:26.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.315+0000 7f156820b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15600c6170 con 0x7f1560072110 2023-12-04T05:40:26.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.315+0000 7f1565fa7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1560072110 0x7f1560158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:26.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.315+0000 7f15667a8640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f15600c56f0 0x7f1560140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40262/0 (socket says 172.21.15.107:40262) 2023-12-04T05:40:26.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.315+0000 7f15667a8640 1 -- 172.21.15.107:0/2211769822 learned_addr learned my addr 172.21.15.107:0/2211769822 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:26.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.315+0000 7f1565fa7640 1 -- 172.21.15.107:0/2211769822 >> v1:172.21.15.195:6789/0 conn(0x7f15600724f0 legacy=0x7f15600c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:26.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.315+0000 7f1565fa7640 1 -- 172.21.15.107:0/2211769822 >> v1:172.21.15.107:6789/0 conn(0x7f15600c56f0 legacy=0x7f1560140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:26.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.315+0000 7f1565fa7640 1 -- 172.21.15.107:0/2211769822 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1560157da0 con 0x7f1560072110 2023-12-04T05:40:26.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.315+0000 7f1565fa7640 1 --2- 172.21.15.107:0/2211769822 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1560072110 0x7f1560158860 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f155c0097c0 tx=0x7f155c01ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cff289ff38e0ad5 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:26.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.315+0000 7f1564fa5640 1 -- 172.21.15.107:0/2211769822 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155c0a8930 con 0x7f1560072110 2023-12-04T05:40:26.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.315+0000 7f1564fa5640 1 -- 172.21.15.107:0/2211769822 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155c0a8a90 con 0x7f1560072110 2023-12-04T05:40:26.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.315+0000 7f1564fa5640 1 -- 172.21.15.107:0/2211769822 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155c0a8e20 con 0x7f1560072110 2023-12-04T05:40:26.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.315+0000 7f156820b640 1 -- 172.21.15.107:0/2211769822 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1560072110 msgr2=0x7f1560158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:26.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.315+0000 7f156820b640 1 --2- 172.21.15.107:0/2211769822 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1560072110 0x7f1560158860 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f155c0097c0 tx=0x7f155c01ec30 comp rx=0 tx=0).stop 2023-12-04T05:40:26.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f156820b640 1 -- 172.21.15.107:0/2211769822 shutdown_connections 2023-12-04T05:40:26.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f156820b640 1 --2- 172.21.15.107:0/2211769822 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1560072110 0x7f1560158860 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f156820b640 1 -- 172.21.15.107:0/2211769822 >> 172.21.15.107:0/2211769822 conn(0x7f15600c7ff0 msgr2=0x7f15600c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:26.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f156820b640 1 -- 172.21.15.107:0/2211769822 shutdown_connections 2023-12-04T05:40:26.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f156820b640 1 -- 172.21.15.107:0/2211769822 wait complete. 2023-12-04T05:40:26.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f156820b640 1 Processor -- start 2023-12-04T05:40:26.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f156820b640 1 -- start start 2023-12-04T05:40:26.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f156820b640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1560072110 0x7f15601f0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:26.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f156820b640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15600724f0 0x7f156015abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:26.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f156820b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15600c56f0 0x7f156015b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:26.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f156820b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15601f1be0 con 0x7f15600c56f0 2023-12-04T05:40:26.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f156820b640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15600b9260 con 0x7f1560072110 2023-12-04T05:40:26.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f156820b640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15600ca440 con 0x7f15600724f0 2023-12-04T05:40:26.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f15667a8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15600c56f0 0x7f156015b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:26.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f15667a8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15600c56f0 0x7f156015b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53086/0 (socket says 172.21.15.107:53086) 2023-12-04T05:40:26.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f15667a8640 1 -- 172.21.15.107:0/1421509265 learned_addr learned my addr 172.21.15.107:0/1421509265 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:26.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f15667a8640 1 -- 172.21.15.107:0/1421509265 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15600724f0 msgr2=0x7f156015abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:26.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f15667a8640 1 --2- 172.21.15.107:0/1421509265 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15600724f0 0x7f156015abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f15667a8640 1 -- 172.21.15.107:0/1421509265 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1560072110 msgr2=0x7f15601f0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:26.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f1565fa7640 1 --2- 172.21.15.107:0/1421509265 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1560072110 0x7f15601f0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:26.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f15667a8640 1 --2- 172.21.15.107:0/1421509265 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1560072110 0x7f15601f0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f15667a8640 1 -- 172.21.15.107:0/1421509265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155c009420 con 0x7f15600c56f0 2023-12-04T05:40:26.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f15667a8640 1 --2- 172.21.15.107:0/1421509265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15600c56f0 0x7f156015b110 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f155800cab0 tx=0x7f155800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:26.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f15567fc640 1 -- 172.21.15.107:0/1421509265 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155800ac30 con 0x7f15600c56f0 2023-12-04T05:40:26.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f15567fc640 1 -- 172.21.15.107:0/1421509265 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155800ad90 con 0x7f15600c56f0 2023-12-04T05:40:26.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f156820b640 1 -- 172.21.15.107:0/1421509265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15600c9fb0 con 0x7f15600c56f0 2023-12-04T05:40:26.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f15567fc640 1 -- 172.21.15.107:0/1421509265 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1558012600 con 0x7f15600c56f0 2023-12-04T05:40:26.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.319+0000 7f156820b640 1 -- 172.21.15.107:0/1421509265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15601f1900 con 0x7f15600c56f0 2023-12-04T05:40:26.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.323+0000 7f156820b640 1 -- 172.21.15.107:0/1421509265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15600ca140 con 0x7f15600c56f0 2023-12-04T05:40:26.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.323+0000 7f15567fc640 1 -- 172.21.15.107:0/1421509265 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f1558012760 con 0x7f15600c56f0 2023-12-04T05:40:26.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.323+0000 7f15567fc640 1 --2- 172.21.15.107:0/1421509265 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f15440668d0 0x7f1544068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:26.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.323+0000 7f1565fa7640 1 --2- 172.21.15.107:0/1421509265 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f15440668d0 0x7f1544068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:26.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.323+0000 7f15567fc640 1 -- 172.21.15.107:0/1421509265 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1558084070 con 0x7f15600c56f0 2023-12-04T05:40:26.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.323+0000 7f1565fa7640 1 --2- 172.21.15.107:0/1421509265 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f15440668d0 0x7f1544068d90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f155c0b2000 tx=0x7f155c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:26.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.327+0000 7f15567fc640 1 -- 172.21.15.107:0/1421509265 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1558051360 con 0x7f15600c56f0 2023-12-04T05:40:26.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.479+0000 7f156820b640 1 -- 172.21.15.107:0/1421509265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1560158da0 con 0x7f15600c56f0 2023-12-04T05:40:26.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.479+0000 7f15567fc640 1 -- 172.21.15.107:0/1421509265 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1560158da0 con 0x7f15600c56f0 2023-12-04T05:40:26.482 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-12-04T05:40:26.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.483+0000 7f156820b640 1 -- 172.21.15.107:0/1421509265 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f15440668d0 msgr2=0x7f1544068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:26.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.483+0000 7f156820b640 1 --2- 172.21.15.107:0/1421509265 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f15440668d0 0x7f1544068d90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f155c0b2000 tx=0x7f155c008040 comp rx=0 tx=0).stop 2023-12-04T05:40:26.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.483+0000 7f156820b640 1 -- 172.21.15.107:0/1421509265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15600c56f0 msgr2=0x7f156015b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:26.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.483+0000 7f156820b640 1 --2- 172.21.15.107:0/1421509265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15600c56f0 0x7f156015b110 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f155800cab0 tx=0x7f155800cf70 comp rx=0 tx=0).stop 2023-12-04T05:40:26.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.483+0000 7f156820b640 1 -- 172.21.15.107:0/1421509265 shutdown_connections 2023-12-04T05:40:26.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.483+0000 7f156820b640 1 --2- 172.21.15.107:0/1421509265 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f15440668d0 0x7f1544068d90 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.483+0000 7f156820b640 1 --2- 172.21.15.107:0/1421509265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15600c56f0 0x7f156015b110 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.483+0000 7f156820b640 1 --2- 172.21.15.107:0/1421509265 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15600724f0 0x7f156015abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.483+0000 7f156820b640 1 --2- 172.21.15.107:0/1421509265 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1560072110 0x7f15601f0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.483+0000 7f156820b640 1 -- 172.21.15.107:0/1421509265 >> 172.21.15.107:0/1421509265 conn(0x7f15600c7ff0 msgr2=0x7f15600c71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:26.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.483+0000 7f156820b640 1 -- 172.21.15.107:0/1421509265 shutdown_connections 2023-12-04T05:40:26.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.483+0000 7f156820b640 1 -- 172.21.15.107:0/1421509265 wait complete. 2023-12-04T05:40:26.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-04T05:40:26.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.619+0000 7fb9a3b2b640 1 Processor -- start 2023-12-04T05:40:26.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.619+0000 7fb9a3b2b640 1 -- start start 2023-12-04T05:40:26.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a3b2b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb99c072110 0x7fb99c158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a3b2b640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb99c0724f0 0x7fb99c0c7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a3b2b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb99c158da0 con 0x7fb99c0c56f0 2023-12-04T05:40:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a3b2b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb99c1583d0 con 0x7fb99c072110 2023-12-04T05:40:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a3b2b640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c0c6170 con 0x7fb99c0724f0 2023-12-04T05:40:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a18c7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb99c072110 0x7fb99c158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a18c7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb99c072110 0x7fb99c158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53102/0 (socket says 172.21.15.107:53102) 2023-12-04T05:40:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a18c7640 1 -- 172.21.15.107:0/2590549997 learned_addr learned my addr 172.21.15.107:0/2590549997 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a10c6640 1 --2- 172.21.15.107:0/2590549997 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb99c0724f0 0x7fb99c0c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a18c7640 1 -- 172.21.15.107:0/2590549997 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb99c0724f0 msgr2=0x7fb99c0c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a18c7640 1 --2- 172.21.15.107:0/2590549997 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb99c0724f0 0x7fb99c0c7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a18c7640 1 -- 172.21.15.107:0/2590549997 >> v1:172.21.15.195:6789/0 conn(0x7fb99c0c56f0 legacy=0x7fb99c140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:26.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a18c7640 1 -- 172.21.15.107:0/2590549997 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb99c157da0 con 0x7fb99c072110 2023-12-04T05:40:26.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a18c7640 1 --2- 172.21.15.107:0/2590549997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb99c072110 0x7fb99c158860 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb98c009870 tx=0x7fb98c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b86028508072754 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:26.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a08c5640 1 -- 172.21.15.107:0/2590549997 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb98c0a89b0 con 0x7fb99c072110 2023-12-04T05:40:26.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a08c5640 1 -- 172.21.15.107:0/2590549997 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb98c0a8b10 con 0x7fb99c072110 2023-12-04T05:40:26.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a08c5640 1 -- 172.21.15.107:0/2590549997 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb98c0a8ea0 con 0x7fb99c072110 2023-12-04T05:40:26.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a3b2b640 1 -- 172.21.15.107:0/2590549997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb99c072110 msgr2=0x7fb99c158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:26.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a3b2b640 1 --2- 172.21.15.107:0/2590549997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb99c072110 0x7fb99c158860 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb98c009870 tx=0x7fb98c01ecf0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a3b2b640 1 -- 172.21.15.107:0/2590549997 shutdown_connections 2023-12-04T05:40:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a3b2b640 1 --2- 172.21.15.107:0/2590549997 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb99c0724f0 0x7fb99c0c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a3b2b640 1 --2- 172.21.15.107:0/2590549997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb99c072110 0x7fb99c158860 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a3b2b640 1 -- 172.21.15.107:0/2590549997 >> 172.21.15.107:0/2590549997 conn(0x7fb99c0c7ff0 msgr2=0x7fb99c0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a3b2b640 1 -- 172.21.15.107:0/2590549997 shutdown_connections 2023-12-04T05:40:26.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.623+0000 7fb9a3b2b640 1 -- 172.21.15.107:0/2590549997 wait complete. 2023-12-04T05:40:26.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a3b2b640 1 Processor -- start 2023-12-04T05:40:26.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a3b2b640 1 -- start start 2023-12-04T05:40:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a3b2b640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb99c072110 0x7fb99c14c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a3b2b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb99c0724f0 0x7fb99c0ca9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a3b2b640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb99c0c56f0 0x7fb99c1f0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a3b2b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c14cbb0 con 0x7fb99c0724f0 2023-12-04T05:40:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a3b2b640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c152d60 con 0x7fb99c0c56f0 2023-12-04T05:40:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a3b2b640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c14c1e0 con 0x7fb99c072110 2023-12-04T05:40:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a10c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb99c0724f0 0x7fb99c0ca9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a10c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb99c0724f0 0x7fb99c0ca9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53114/0 (socket says 172.21.15.107:53114) 2023-12-04T05:40:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a10c6640 1 -- 172.21.15.107:0/1362184781 learned_addr learned my addr 172.21.15.107:0/1362184781 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a18c7640 1 --2- 172.21.15.107:0/1362184781 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb99c072110 0x7fb99c14c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a20c8640 1 --2- 172.21.15.107:0/1362184781 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb99c0c56f0 0x7fb99c1f0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a10c6640 1 -- 172.21.15.107:0/1362184781 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb99c072110 msgr2=0x7fb99c14c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a10c6640 1 --2- 172.21.15.107:0/1362184781 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb99c072110 0x7fb99c14c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a10c6640 1 -- 172.21.15.107:0/1362184781 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb99c0c56f0 msgr2=0x7fb99c1f0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a10c6640 1 --2- 172.21.15.107:0/1362184781 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb99c0c56f0 0x7fb99c1f0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a10c6640 1 -- 172.21.15.107:0/1362184781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb98c0094d0 con 0x7fb99c0724f0 2023-12-04T05:40:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a20c8640 1 --2- 172.21.15.107:0/1362184781 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb99c0c56f0 0x7fb99c1f0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a10c6640 1 --2- 172.21.15.107:0/1362184781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb99c0724f0 0x7fb99c0ca9c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb99800d800 tx=0x7fb99800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9927fc640 1 -- 172.21.15.107:0/1362184781 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb99800b6c0 con 0x7fb99c0724f0 2023-12-04T05:40:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9927fc640 1 -- 172.21.15.107:0/1362184781 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb99800b820 con 0x7fb99c0724f0 2023-12-04T05:40:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a3b2b640 1 -- 172.21.15.107:0/1362184781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb99c152f90 con 0x7fb99c0724f0 2023-12-04T05:40:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9927fc640 1 -- 172.21.15.107:0/1362184781 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb99800bb90 con 0x7fb99c0724f0 2023-12-04T05:40:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a3b2b640 1 -- 172.21.15.107:0/1362184781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb99c14bb90 con 0x7fb99c0724f0 2023-12-04T05:40:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.627+0000 7fb9a3b2b640 1 -- 172.21.15.107:0/1362184781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb99c14b9f0 con 0x7fb99c0724f0 2023-12-04T05:40:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.631+0000 7fb9927fc640 1 -- 172.21.15.107:0/1362184781 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fb99800bcf0 con 0x7fb99c0724f0 2023-12-04T05:40:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.631+0000 7fb9927fc640 1 --2- 172.21.15.107:0/1362184781 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb9800668d0 0x7fb980068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.631+0000 7fb9927fc640 1 -- 172.21.15.107:0/1362184781 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb998084970 con 0x7fb99c0724f0 2023-12-04T05:40:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.631+0000 7fb9a18c7640 1 --2- 172.21.15.107:0/1362184781 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb9800668d0 0x7fb980068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.631+0000 7fb9a18c7640 1 --2- 172.21.15.107:0/1362184781 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb9800668d0 0x7fb980068d90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb98c0b2000 tx=0x7fb98c0aa070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:26.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.635+0000 7fb9927fc640 1 -- 172.21.15.107:0/1362184781 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb99c14b9f0 con 0x7fb99c0724f0 2023-12-04T05:40:26.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.775+0000 7fb9a3b2b640 1 -- 172.21.15.107:0/1362184781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb99c14b9f0 con 0x7fb99c0724f0 2023-12-04T05:40:26.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.775+0000 7fb9927fc640 1 -- 172.21.15.107:0/1362184781 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb99c14b9f0 con 0x7fb99c0724f0 2023-12-04T05:40:26.779 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-12-04T05:40:26.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.779+0000 7fb9a3b2b640 1 -- 172.21.15.107:0/1362184781 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb9800668d0 msgr2=0x7fb980068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:26.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.779+0000 7fb9a3b2b640 1 --2- 172.21.15.107:0/1362184781 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb9800668d0 0x7fb980068d90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb98c0b2000 tx=0x7fb98c0aa070 comp rx=0 tx=0).stop 2023-12-04T05:40:26.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.779+0000 7fb9a3b2b640 1 -- 172.21.15.107:0/1362184781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb99c0724f0 msgr2=0x7fb99c0ca9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:26.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.779+0000 7fb9a3b2b640 1 --2- 172.21.15.107:0/1362184781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb99c0724f0 0x7fb99c0ca9c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb99800d800 tx=0x7fb99800dcc0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.779+0000 7fb9a3b2b640 1 -- 172.21.15.107:0/1362184781 shutdown_connections 2023-12-04T05:40:26.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.779+0000 7fb9a3b2b640 1 --2- 172.21.15.107:0/1362184781 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb9800668d0 0x7fb980068d90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.779+0000 7fb9a3b2b640 1 --2- 172.21.15.107:0/1362184781 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb99c0c56f0 0x7fb99c1f0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.779+0000 7fb9a3b2b640 1 --2- 172.21.15.107:0/1362184781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb99c0724f0 0x7fb99c0ca9c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.779+0000 7fb9a3b2b640 1 --2- 172.21.15.107:0/1362184781 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb99c072110 0x7fb99c14c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.779+0000 7fb9a3b2b640 1 -- 172.21.15.107:0/1362184781 >> 172.21.15.107:0/1362184781 conn(0x7fb99c0c7ff0 msgr2=0x7fb99c0c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:26.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.779+0000 7fb9a3b2b640 1 -- 172.21.15.107:0/1362184781 shutdown_connections 2023-12-04T05:40:26.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.779+0000 7fb9a3b2b640 1 -- 172.21.15.107:0/1362184781 wait complete. 2023-12-04T05:40:26.797 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-04T05:40:26.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.914+0000 7f2383990640 1 Processor -- start 2023-12-04T05:40:26.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.914+0000 7f2383990640 1 -- start start 2023-12-04T05:40:26.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.918+0000 7f2383990640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f237c0724f0 0x7f237c158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:26.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.918+0000 7f2383990640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c158da0 con 0x7f237c0c56f0 2023-12-04T05:40:26.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.918+0000 7f2383990640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c1583d0 con 0x7f237c072110 2023-12-04T05:40:26.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.918+0000 7f2383990640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f237c0c6170 con 0x7f237c0724f0 2023-12-04T05:40:26.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.918+0000 7f2380f2b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f237c0724f0 0x7f237c158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:26.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.918+0000 7f2381f2d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f237c0c56f0 0x7f237c140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40266/0 (socket says 172.21.15.107:40266) 2023-12-04T05:40:26.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.918+0000 7f2381f2d640 1 -- 172.21.15.107:0/2559081999 learned_addr learned my addr 172.21.15.107:0/2559081999 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:26.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.918+0000 7f2380f2b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f237c0724f0 0x7f237c158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53128/0 (socket says 172.21.15.107:53128) 2023-12-04T05:40:26.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.918+0000 7f2380f2b640 1 -- 172.21.15.107:0/2559081999 >> v1:172.21.15.195:6789/0 conn(0x7f237c072110 legacy=0x7f237c0c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:26.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.918+0000 7f2380f2b640 1 -- 172.21.15.107:0/2559081999 >> v1:172.21.15.107:6789/0 conn(0x7f237c0c56f0 legacy=0x7f237c140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:26.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.918+0000 7f2373fff640 1 -- 172.21.15.107:0/2559081999 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189445747 0 0) 0x7f23740014a0 con 0x7f237c0c56f0 2023-12-04T05:40:26.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.918+0000 7f2380f2b640 1 -- 172.21.15.107:0/2559081999 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c157da0 con 0x7f237c0724f0 2023-12-04T05:40:26.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.918+0000 7f2380f2b640 1 --2- 172.21.15.107:0/2559081999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f237c0724f0 0x7f237c158860 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f236c0097c0 tx=0x7f236c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25e99f560211bb0c server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:26.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.918+0000 7f2373fff640 1 -- 172.21.15.107:0/2559081999 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236c0a8970 con 0x7f237c0724f0 2023-12-04T05:40:26.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.918+0000 7f2373fff640 1 -- 172.21.15.107:0/2559081999 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236c0a8ad0 con 0x7f237c0724f0 2023-12-04T05:40:26.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.918+0000 7f2373fff640 1 -- 172.21.15.107:0/2559081999 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236c0a8e60 con 0x7f237c0724f0 2023-12-04T05:40:26.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.918+0000 7f2383990640 1 -- 172.21.15.107:0/2559081999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f237c0724f0 msgr2=0x7f237c158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:26.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.918+0000 7f2383990640 1 --2- 172.21.15.107:0/2559081999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f237c0724f0 0x7f237c158860 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f236c0097c0 tx=0x7f236c01ec70 comp rx=0 tx=0).stop 2023-12-04T05:40:26.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.918+0000 7f2383990640 1 -- 172.21.15.107:0/2559081999 shutdown_connections 2023-12-04T05:40:26.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.918+0000 7f2383990640 1 --2- 172.21.15.107:0/2559081999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f237c0724f0 0x7f237c158860 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.918+0000 7f2383990640 1 -- 172.21.15.107:0/2559081999 >> 172.21.15.107:0/2559081999 conn(0x7f237c0c7ff0 msgr2=0x7f237c0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:26.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.922+0000 7f2383990640 1 -- 172.21.15.107:0/2559081999 shutdown_connections 2023-12-04T05:40:26.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.922+0000 7f2383990640 1 -- 172.21.15.107:0/2559081999 wait complete. 2023-12-04T05:40:26.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.922+0000 7f2383990640 1 Processor -- start 2023-12-04T05:40:26.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.922+0000 7f2383990640 1 -- start start 2023-12-04T05:40:26.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.922+0000 7f2383990640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f237c072110 0x7f237c159880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:26.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.922+0000 7f2383990640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f237c0724f0 0x7f237c1f1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:26.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.922+0000 7f2383990640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f237c0c56f0 0x7f237c0ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:26.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.922+0000 7f2383990640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c1f0b80 con 0x7f237c0724f0 2023-12-04T05:40:26.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.922+0000 7f2383990640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c159dc0 con 0x7f237c0c56f0 2023-12-04T05:40:26.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.922+0000 7f2383990640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237c1f1ad0 con 0x7f237c072110 2023-12-04T05:40:26.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.922+0000 7f2380f2b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f237c0724f0 0x7f237c1f1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:26.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.922+0000 7f2380f2b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f237c0724f0 0x7f237c1f1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53138/0 (socket says 172.21.15.107:53138) 2023-12-04T05:40:26.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.922+0000 7f2380f2b640 1 -- 172.21.15.107:0/238561250 learned_addr learned my addr 172.21.15.107:0/238561250 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:26.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.922+0000 7f2381f2d640 1 --2- 172.21.15.107:0/238561250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f237c0c56f0 0x7f237c0ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:26.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.922+0000 7f238172c640 1 --2- 172.21.15.107:0/238561250 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f237c072110 0x7f237c159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:26.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.922+0000 7f2380f2b640 1 -- 172.21.15.107:0/238561250 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f237c072110 msgr2=0x7f237c159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:26.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.922+0000 7f2380f2b640 1 --2- 172.21.15.107:0/238561250 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f237c072110 0x7f237c159880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.922+0000 7f2380f2b640 1 -- 172.21.15.107:0/238561250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f237c0c56f0 msgr2=0x7f237c0ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:26.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.922+0000 7f2380f2b640 1 --2- 172.21.15.107:0/238561250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f237c0c56f0 0x7f237c0ca6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:26.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.922+0000 7f2380f2b640 1 -- 172.21.15.107:0/238561250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f236c009420 con 0x7f237c0724f0 2023-12-04T05:40:26.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.922+0000 7f2380f2b640 1 --2- 172.21.15.107:0/238561250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f237c0724f0 0x7f237c1f1590 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f236c0097c0 tx=0x7f236c01ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:26.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.926+0000 7f2371ffb640 1 -- 172.21.15.107:0/238561250 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236c0b9630 con 0x7f237c0724f0 2023-12-04T05:40:26.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.926+0000 7f2371ffb640 1 -- 172.21.15.107:0/238561250 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236c0b97b0 con 0x7f237c0724f0 2023-12-04T05:40:26.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.926+0000 7f2383990640 1 -- 172.21.15.107:0/238561250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f237c159fe0 con 0x7f237c0724f0 2023-12-04T05:40:26.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.926+0000 7f2371ffb640 1 -- 172.21.15.107:0/238561250 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236c0b9b60 con 0x7f237c0724f0 2023-12-04T05:40:26.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.926+0000 7f2383990640 1 -- 172.21.15.107:0/238561250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f237c15a5a0 con 0x7f237c0724f0 2023-12-04T05:40:26.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.926+0000 7f2383990640 1 -- 172.21.15.107:0/238561250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f237c15a790 con 0x7f237c0724f0 2023-12-04T05:40:26.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.926+0000 7f2371ffb640 1 -- 172.21.15.107:0/238561250 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f236c0b7040 con 0x7f237c0724f0 2023-12-04T05:40:26.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.926+0000 7f2371ffb640 1 --2- 172.21.15.107:0/238561250 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f23600668d0 0x7f2360068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:26.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.926+0000 7f2371ffb640 1 -- 172.21.15.107:0/238561250 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f236c122560 con 0x7f237c0724f0 2023-12-04T05:40:26.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.926+0000 7f238172c640 1 --2- 172.21.15.107:0/238561250 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f23600668d0 0x7f2360068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:26.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.926+0000 7f238172c640 1 --2- 172.21.15.107:0/238561250 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f23600668d0 0x7f2360068d90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2378000f70 tx=0x7f2378009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:26.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:26.930+0000 7f2371ffb640 1 -- 172.21.15.107:0/238561250 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f236c0ef670 con 0x7f237c0724f0 2023-12-04T05:40:27.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.070+0000 7f2383990640 1 -- 172.21.15.107:0/238561250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f237c067870 con 0x7f237c0724f0 2023-12-04T05:40:27.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.070+0000 7f2371ffb640 1 -- 172.21.15.107:0/238561250 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f236c0b5020 con 0x7f237c0724f0 2023-12-04T05:40:27.076 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-12-04T05:40:27.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.074+0000 7f2383990640 1 -- 172.21.15.107:0/238561250 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f23600668d0 msgr2=0x7f2360068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:27.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.074+0000 7f2383990640 1 --2- 172.21.15.107:0/238561250 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f23600668d0 0x7f2360068d90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2378000f70 tx=0x7f2378009040 comp rx=0 tx=0).stop 2023-12-04T05:40:27.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.074+0000 7f2383990640 1 -- 172.21.15.107:0/238561250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f237c0724f0 msgr2=0x7f237c1f1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:27.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.074+0000 7f2383990640 1 --2- 172.21.15.107:0/238561250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f237c0724f0 0x7f237c1f1590 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f236c0097c0 tx=0x7f236c01ba10 comp rx=0 tx=0).stop 2023-12-04T05:40:27.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.074+0000 7f2383990640 1 -- 172.21.15.107:0/238561250 shutdown_connections 2023-12-04T05:40:27.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.074+0000 7f2383990640 1 --2- 172.21.15.107:0/238561250 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f23600668d0 0x7f2360068d90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.074+0000 7f2383990640 1 --2- 172.21.15.107:0/238561250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f237c0c56f0 0x7f237c0ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.074+0000 7f2383990640 1 --2- 172.21.15.107:0/238561250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f237c0724f0 0x7f237c1f1590 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.074+0000 7f2383990640 1 --2- 172.21.15.107:0/238561250 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f237c072110 0x7f237c159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.074+0000 7f2383990640 1 -- 172.21.15.107:0/238561250 >> 172.21.15.107:0/238561250 conn(0x7f237c0c7ff0 msgr2=0x7f237c156ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:27.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.074+0000 7f2383990640 1 -- 172.21.15.107:0/238561250 shutdown_connections 2023-12-04T05:40:27.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.074+0000 7f2383990640 1 -- 172.21.15.107:0/238561250 wait complete. 2023-12-04T05:40:27.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:40:27.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.210+0000 7f0779b1b640 1 Processor -- start 2023-12-04T05:40:27.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f0779b1b640 1 -- start start 2023-12-04T05:40:27.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f0779b1b640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f07740c46a0 0x7f07740c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f0779b1b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f077406e660 con 0x7f0774157180 2023-12-04T05:40:27.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f0779b1b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0774156cf0 con 0x7f07740c4a80 2023-12-04T05:40:27.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f0779b1b640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0774156490 con 0x7f07740c46a0 2023-12-04T05:40:27.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f076bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0774157180 0x7f07740c5ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40280/0 (socket says 172.21.15.107:40280) 2023-12-04T05:40:27.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f076bfff640 1 -- 172.21.15.107:0/1929734656 learned_addr learned my addr 172.21.15.107:0/1929734656 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:27.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f07737fe640 1 --2- 172.21.15.107:0/1929734656 >> v2:172.21.15.195:3300/0 conn(0x7f07740c46a0 0x7f07740c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:27.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f0772ffd640 1 -- 172.21.15.107:0/1929734656 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726790576 0 0) 0x7f075c001120 con 0x7f0774157180 2023-12-04T05:40:27.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f0772ffd640 1 -- 172.21.15.107:0/1929734656 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0760004640 con 0x7f0774157180 2023-12-04T05:40:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f0772ffd640 1 -- 172.21.15.107:0/1929734656 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 874558853 0 0) 0x7f0760004640 con 0x7f0774157180 2023-12-04T05:40:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f0772ffd640 1 -- 172.21.15.107:0/1929734656 >> v2:172.21.15.195:3300/0 conn(0x7f07740c46a0 msgr2=0x7f07740c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f0772ffd640 1 --2- 172.21.15.107:0/1929734656 >> v2:172.21.15.195:3300/0 conn(0x7f07740c46a0 0x7f07740c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f0772ffd640 1 -- 172.21.15.107:0/1929734656 >> v1:172.21.15.195:6789/0 conn(0x7f07740c4a80 legacy=0x7f07740c50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f0772ffd640 1 -- 172.21.15.107:0/1929734656 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f077406e0a0 con 0x7f0774157180 2023-12-04T05:40:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f0772ffd640 1 -- 172.21.15.107:0/1929734656 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f075c001810 con 0x7f0774157180 2023-12-04T05:40:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f0772ffd640 1 -- 172.21.15.107:0/1929734656 >> v1:172.21.15.107:6789/0 conn(0x7f0774157180 legacy=0x7f07740c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f0772ffd640 1 --2- 172.21.15.107:0/1929734656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0760004a90 0x7f0760004eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f0772ffd640 1 --2- 172.21.15.107:0/1929734656 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f07600058e0 0x7f0760005d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f0772ffd640 1 --2- 172.21.15.107:0/1929734656 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f07600062a0 0x7f076000caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f0772ffd640 1 -- 172.21.15.107:0/1929734656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0774156490 con 0x7f0760004a90 2023-12-04T05:40:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f0772ffd640 1 -- 172.21.15.107:0/1929734656 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076000d140 con 0x7f07600062a0 2023-12-04T05:40:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f0772ffd640 1 -- 172.21.15.107:0/1929734656 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f076000d300 con 0x7f07600058e0 2023-12-04T05:40:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f07737fe640 1 --2- 172.21.15.107:0/1929734656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0760004a90 0x7f0760004eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f07737fe640 1 -- 172.21.15.107:0/1929734656 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f07600058e0 msgr2=0x7f0760005d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f07737fe640 1 --2- 172.21.15.107:0/1929734656 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f07600058e0 0x7f0760005d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f07737fe640 1 -- 172.21.15.107:0/1929734656 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f07600062a0 msgr2=0x7f076000caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:27.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f07737fe640 1 --2- 172.21.15.107:0/1929734656 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f07600062a0 0x7f076000caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f0773fff640 1 -- 172.21.15.107:0/1929734656 reap_dead start 2023-12-04T05:40:27.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.214+0000 7f07737fe640 1 -- 172.21.15.107:0/1929734656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0760004640 con 0x7f0760004a90 2023-12-04T05:40:27.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f07737fe640 1 --2- 172.21.15.107:0/1929734656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0760004a90 0x7f0760004eb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0764009670 tx=0x7f076401eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18c2e02ee10d35a3 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:27.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f0772ffd640 1 -- 172.21.15.107:0/1929734656 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07640a7680 con 0x7f0760004a90 2023-12-04T05:40:27.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f0772ffd640 1 -- 172.21.15.107:0/1929734656 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07640a77e0 con 0x7f0760004a90 2023-12-04T05:40:27.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f0772ffd640 1 -- 172.21.15.107:0/1929734656 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07640a7b90 con 0x7f0760004a90 2023-12-04T05:40:27.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f0779b1b640 1 -- 172.21.15.107:0/1929734656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0760004a90 msgr2=0x7f0760004eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:27.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f0779b1b640 1 --2- 172.21.15.107:0/1929734656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0760004a90 0x7f0760004eb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0764009670 tx=0x7f076401eca0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f0779b1b640 1 -- 172.21.15.107:0/1929734656 shutdown_connections 2023-12-04T05:40:27.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f0779b1b640 1 --2- 172.21.15.107:0/1929734656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0760004a90 0x7f0760004eb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f0779b1b640 1 -- 172.21.15.107:0/1929734656 >> 172.21.15.107:0/1929734656 conn(0x7f07740c7ff0 msgr2=0x7f07740c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:27.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f0779b1b640 1 -- 172.21.15.107:0/1929734656 shutdown_connections 2023-12-04T05:40:27.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f0779b1b640 1 -- 172.21.15.107:0/1929734656 wait complete. 2023-12-04T05:40:27.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f0779b1b640 1 Processor -- start 2023-12-04T05:40:27.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f0779b1b640 1 -- start start 2023-12-04T05:40:27.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f0779b1b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0760004a90 0x7f0774154430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f0779b1b640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f077414daa0 0x7f077414df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f0779b1b640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0774150e00 0x7f0774151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f0779b1b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07741ee4a0 con 0x7f0760004a90 2023-12-04T05:40:27.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f0779b1b640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077414e440 con 0x7f0774150e00 2023-12-04T05:40:27.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f0779b1b640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07741517c0 con 0x7f077414daa0 2023-12-04T05:40:27.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f07737fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0760004a90 0x7f0774154430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:27.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f07737fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0760004a90 0x7f0774154430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53150/0 (socket says 172.21.15.107:53150) 2023-12-04T05:40:27.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f07737fe640 1 -- 172.21.15.107:0/37239491 learned_addr learned my addr 172.21.15.107:0/37239491 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:27.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f07737fe640 1 -- 172.21.15.107:0/37239491 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f077414daa0 msgr2=0x7f077414df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:27.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f07737fe640 1 --2- 172.21.15.107:0/37239491 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f077414daa0 0x7f077414df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f07737fe640 1 -- 172.21.15.107:0/37239491 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0774150e00 msgr2=0x7f0774151280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:27.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f07737fe640 1 --2- 172.21.15.107:0/37239491 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0774150e00 0x7f0774151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f07737fe640 1 -- 172.21.15.107:0/37239491 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07741ee690 con 0x7f0760004a90 2023-12-04T05:40:27.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.218+0000 7f07737fe640 1 --2- 172.21.15.107:0/37239491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0760004a90 0x7f0774154430 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f07640b2040 tx=0x7f076401ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:27.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.222+0000 7f0770ff9640 1 -- 172.21.15.107:0/37239491 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076400f000 con 0x7f0760004a90 2023-12-04T05:40:27.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.222+0000 7f0770ff9640 1 -- 172.21.15.107:0/37239491 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0764004020 con 0x7f0760004a90 2023-12-04T05:40:27.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.222+0000 7f0779b1b640 1 -- 172.21.15.107:0/37239491 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07741ee970 con 0x7f0760004a90 2023-12-04T05:40:27.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.222+0000 7f0770ff9640 1 -- 172.21.15.107:0/37239491 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764004390 con 0x7f0760004a90 2023-12-04T05:40:27.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.222+0000 7f0779b1b640 1 -- 172.21.15.107:0/37239491 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07741eef30 con 0x7f0760004a90 2023-12-04T05:40:27.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.222+0000 7f0779b1b640 1 -- 172.21.15.107:0/37239491 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0774067870 con 0x7f0760004a90 2023-12-04T05:40:27.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.222+0000 7f0770ff9640 1 -- 172.21.15.107:0/37239491 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f07640b6070 con 0x7f0760004a90 2023-12-04T05:40:27.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.222+0000 7f0770ff9640 1 --2- 172.21.15.107:0/37239491 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f07540668d0 0x7f0754068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.222+0000 7f0770ff9640 1 -- 172.21.15.107:0/37239491 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f07641205a0 con 0x7f0760004a90 2023-12-04T05:40:27.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.222+0000 7f076bfff640 1 --2- 172.21.15.107:0/37239491 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f07540668d0 0x7f0754068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:27.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.222+0000 7f076bfff640 1 --2- 172.21.15.107:0/37239491 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f07540668d0 0x7f0754068d90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f075c000f70 tx=0x7f075c001ba0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.226+0000 7f0770ff9640 1 -- 172.21.15.107:0/37239491 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f07640ed870 con 0x7f0760004a90 2023-12-04T05:40:27.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.414+0000 7f0779b1b640 1 -- 172.21.15.107:0/37239491 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0774152ce0 con 0x7f0760004a90 2023-12-04T05:40:27.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.414+0000 7f0770ff9640 1 -- 172.21.15.107:0/37239491 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+6564 (secure 0 0 0) 0x7f07640f0e90 con 0x7f0760004a90 2023-12-04T05:40:27.418 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:27.419 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:40:23.915736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":16,"state":"up:creating","state_seq":1,"addr":"172.21.15.195:6836/3769196693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":3769196693},{"type":"v1","addr":"172.21.15.195:6836","nonce":3769196693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:40:27.419 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 16 2023-12-04T05:40:27.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.418+0000 7f0779b1b640 1 -- 172.21.15.107:0/37239491 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f07540668d0 msgr2=0x7f0754068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:27.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.418+0000 7f0779b1b640 1 --2- 172.21.15.107:0/37239491 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f07540668d0 0x7f0754068d90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f075c000f70 tx=0x7f075c001ba0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.418+0000 7f0779b1b640 1 -- 172.21.15.107:0/37239491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0760004a90 msgr2=0x7f0774154430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:27.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.418+0000 7f0779b1b640 1 --2- 172.21.15.107:0/37239491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0760004a90 0x7f0774154430 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f07640b2040 tx=0x7f076401ba40 comp rx=0 tx=0).stop 2023-12-04T05:40:27.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.418+0000 7f0779b1b640 1 -- 172.21.15.107:0/37239491 shutdown_connections 2023-12-04T05:40:27.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.418+0000 7f0779b1b640 1 --2- 172.21.15.107:0/37239491 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f07540668d0 0x7f0754068d90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.418+0000 7f0779b1b640 1 --2- 172.21.15.107:0/37239491 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0774150e00 0x7f0774151280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.418+0000 7f0779b1b640 1 --2- 172.21.15.107:0/37239491 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f077414daa0 0x7f077414df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.418+0000 7f0779b1b640 1 --2- 172.21.15.107:0/37239491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0760004a90 0x7f0774154430 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.418+0000 7f0779b1b640 1 -- 172.21.15.107:0/37239491 >> 172.21.15.107:0/37239491 conn(0x7f07740c7ff0 msgr2=0x7f0774155240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:27.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.418+0000 7f0779b1b640 1 -- 172.21.15.107:0/37239491 shutdown_connections 2023-12-04T05:40:27.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.418+0000 7f0779b1b640 1 -- 172.21.15.107:0/37239491 wait complete. 2023-12-04T05:40:27.438 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T05:40:27.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.550+0000 7f05f1bbe640 1 Processor -- start 2023-12-04T05:40:27.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.550+0000 7f05f1bbe640 1 -- start start 2023-12-04T05:40:27.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05f1bbe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f05ec0c4c10 0x7f05ec0c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05f1bbe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05ec15d710 con 0x7f05ec0d01c0 2023-12-04T05:40:27.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05f1bbe640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05ec15d9f0 con 0x7f05ec154020 2023-12-04T05:40:27.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05f1bbe640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05ec0ca230 con 0x7f05ec0c4c10 2023-12-04T05:40:27.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05eaffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f05ec0d01c0 0x7f05ec071ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42310/0 (socket says 172.21.15.107:42310) 2023-12-04T05:40:27.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05eaffd640 1 -- 172.21.15.107:0/844657685 learned_addr learned my addr 172.21.15.107:0/844657685 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:27.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05ebfff640 1 --2- 172.21.15.107:0/844657685 >> v2:172.21.15.195:3300/0 conn(0x7f05ec0c4c10 0x7f05ec0c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:27.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05ea7fc640 1 -- 172.21.15.107:0/844657685 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355198796 0 0) 0x7f05ec15d710 con 0x7f05ec0d01c0 2023-12-04T05:40:27.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05ea7fc640 1 -- 172.21.15.107:0/844657685 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05d8003530 con 0x7f05ec0d01c0 2023-12-04T05:40:27.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05ea7fc640 1 -- 172.21.15.107:0/844657685 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4174106113 0 0) 0x7f05d8003530 con 0x7f05ec0d01c0 2023-12-04T05:40:27.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05ea7fc640 1 -- 172.21.15.107:0/844657685 >> v2:172.21.15.195:3300/0 conn(0x7f05ec0c4c10 msgr2=0x7f05ec0c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:27.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05ea7fc640 1 --2- 172.21.15.107:0/844657685 >> v2:172.21.15.195:3300/0 conn(0x7f05ec0c4c10 0x7f05ec0c5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05ea7fc640 1 -- 172.21.15.107:0/844657685 >> v1:172.21.15.195:6789/0 conn(0x7f05ec154020 legacy=0x7f05ec15a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:27.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05ea7fc640 1 -- 172.21.15.107:0/844657685 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ec15dcd0 con 0x7f05ec0d01c0 2023-12-04T05:40:27.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05ea7fc640 1 -- 172.21.15.107:0/844657685 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f05d4001680 con 0x7f05ec0d01c0 2023-12-04T05:40:27.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05ea7fc640 1 -- 172.21.15.107:0/844657685 >> v1:172.21.15.107:6789/0 conn(0x7f05ec0d01c0 legacy=0x7f05ec071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:27.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05ea7fc640 1 --2- 172.21.15.107:0/844657685 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05d8003980 0x7f05d8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05ea7fc640 1 --2- 172.21.15.107:0/844657685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05d80047d0 0x7f05d8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05ea7fc640 1 --2- 172.21.15.107:0/844657685 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f05d8005190 0x7f05d800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05ea7fc640 1 -- 172.21.15.107:0/844657685 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d800c010 con 0x7f05d80047d0 2023-12-04T05:40:27.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05ea7fc640 1 -- 172.21.15.107:0/844657685 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d800c180 con 0x7f05d8003980 2023-12-04T05:40:27.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05ea7fc640 1 -- 172.21.15.107:0/844657685 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d800c340 con 0x7f05d8005190 2023-12-04T05:40:27.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05eaffd640 1 --2- 172.21.15.107:0/844657685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05d80047d0 0x7f05d8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:27.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05eb7fe640 1 --2- 172.21.15.107:0/844657685 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05d8003980 0x7f05d8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:27.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05ebfff640 1 --2- 172.21.15.107:0/844657685 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f05d8005190 0x7f05d800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:27.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05eaffd640 1 -- 172.21.15.107:0/844657685 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f05d8005190 msgr2=0x7f05d800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:27.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05eaffd640 1 --2- 172.21.15.107:0/844657685 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f05d8005190 0x7f05d800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05eaffd640 1 -- 172.21.15.107:0/844657685 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05d8003980 msgr2=0x7f05d8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:27.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05eaffd640 1 --2- 172.21.15.107:0/844657685 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05d8003980 0x7f05d8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05eaffd640 1 -- 172.21.15.107:0/844657685 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d8003530 con 0x7f05d80047d0 2023-12-04T05:40:27.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05ebfff640 1 -- 172.21.15.107:0/844657685 reap_dead start 2023-12-04T05:40:27.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05eb7fe640 1 --2- 172.21.15.107:0/844657685 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05d8003980 0x7f05d8003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:40:27.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05eaffd640 1 --2- 172.21.15.107:0/844657685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05d80047d0 0x7f05d8004c50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f05d400c1c0 tx=0x7f05d401cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb3330984eade963 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:27.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05ea7fc640 1 -- 172.21.15.107:0/844657685 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d4005ae0 con 0x7f05d80047d0 2023-12-04T05:40:27.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05ea7fc640 1 -- 172.21.15.107:0/844657685 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d4005c40 con 0x7f05d80047d0 2023-12-04T05:40:27.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.554+0000 7f05ea7fc640 1 -- 172.21.15.107:0/844657685 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d40a5ae0 con 0x7f05d80047d0 2023-12-04T05:40:27.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05f1bbe640 1 -- 172.21.15.107:0/844657685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05d80047d0 msgr2=0x7f05d8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:27.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05f1bbe640 1 --2- 172.21.15.107:0/844657685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05d80047d0 0x7f05d8004c50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f05d400c1c0 tx=0x7f05d401cba0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05f1bbe640 1 -- 172.21.15.107:0/844657685 shutdown_connections 2023-12-04T05:40:27.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05f1bbe640 1 --2- 172.21.15.107:0/844657685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05d80047d0 0x7f05d8004c50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05f1bbe640 1 -- 172.21.15.107:0/844657685 >> 172.21.15.107:0/844657685 conn(0x7f05ec06c6c0 msgr2=0x7f05ec0cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:27.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05f1bbe640 1 -- 172.21.15.107:0/844657685 shutdown_connections 2023-12-04T05:40:27.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05f1bbe640 1 -- 172.21.15.107:0/844657685 wait complete. 2023-12-04T05:40:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05f1bbe640 1 Processor -- start 2023-12-04T05:40:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05f1bbe640 1 -- start start 2023-12-04T05:40:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05f1bbe640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05d80047d0 0x7f05ec0b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05f1bbe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05ec0c6a90 0x7f05ec1524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05f1bbe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f05ec15cbf0 0x7f05ec149f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05f1bbe640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ec15c030 con 0x7f05ec0c6a90 2023-12-04T05:40:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05f1bbe640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ec0b4c30 con 0x7f05d80047d0 2023-12-04T05:40:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05f1bbe640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05ec1532f0 con 0x7f05ec15cbf0 2023-12-04T05:40:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05eaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05ec0c6a90 0x7f05ec1524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05eaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05ec0c6a90 0x7f05ec1524e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56284/0 (socket says 172.21.15.107:56284) 2023-12-04T05:40:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05eaffd640 1 -- 172.21.15.107:0/3522863980 learned_addr learned my addr 172.21.15.107:0/3522863980 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05ebfff640 1 --2- 172.21.15.107:0/3522863980 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f05ec15cbf0 0x7f05ec149f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05eb7fe640 1 --2- 172.21.15.107:0/3522863980 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05d80047d0 0x7f05ec0b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05eaffd640 1 -- 172.21.15.107:0/3522863980 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f05ec15cbf0 msgr2=0x7f05ec149f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05eaffd640 1 --2- 172.21.15.107:0/3522863980 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f05ec15cbf0 0x7f05ec149f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05eaffd640 1 -- 172.21.15.107:0/3522863980 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05d80047d0 msgr2=0x7f05ec0b3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05eaffd640 1 --2- 172.21.15.107:0/3522863980 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05d80047d0 0x7f05ec0b3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05eaffd640 1 -- 172.21.15.107:0/3522863980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ec0c81a0 con 0x7f05ec0c6a90 2023-12-04T05:40:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05eb7fe640 1 --2- 172.21.15.107:0/3522863980 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05d80047d0 0x7f05ec0b3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05eaffd640 1 --2- 172.21.15.107:0/3522863980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05ec0c6a90 0x7f05ec1524e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f05d401c610 tx=0x7f05d8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05e3fff640 1 -- 172.21.15.107:0/3522863980 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d40b4070 con 0x7f05ec0c6a90 2023-12-04T05:40:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05e3fff640 1 -- 172.21.15.107:0/3522863980 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d40a58a0 con 0x7f05ec0c6a90 2023-12-04T05:40:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05f1bbe640 1 -- 172.21.15.107:0/3522863980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05ec0b4230 con 0x7f05ec0c6a90 2023-12-04T05:40:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05e3fff640 1 -- 172.21.15.107:0/3522863980 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d40a5c10 con 0x7f05ec0c6a90 2023-12-04T05:40:27.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.558+0000 7f05f1bbe640 1 -- 172.21.15.107:0/3522863980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05ec0b2a30 con 0x7f05ec0c6a90 2023-12-04T05:40:27.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.562+0000 7f05f1bbe640 1 -- 172.21.15.107:0/3522863980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05ec067870 con 0x7f05ec0c6a90 2023-12-04T05:40:27.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.562+0000 7f05e3fff640 1 -- 172.21.15.107:0/3522863980 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f05d4007070 con 0x7f05ec0c6a90 2023-12-04T05:40:27.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.562+0000 7f05e3fff640 1 --2- 172.21.15.107:0/3522863980 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f05cc0668d0 0x7f05cc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.562+0000 7f05eb7fe640 1 --2- 172.21.15.107:0/3522863980 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f05cc0668d0 0x7f05cc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:27.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.562+0000 7f05e3fff640 1 -- 172.21.15.107:0/3522863980 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05d411dec0 con 0x7f05ec0c6a90 2023-12-04T05:40:27.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.562+0000 7f05eb7fe640 1 --2- 172.21.15.107:0/3522863980 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f05cc0668d0 0x7f05cc068d90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f05dc000f90 tx=0x7f05dc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:27.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.566+0000 7f05e3fff640 1 -- 172.21.15.107:0/3522863980 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f05d40eb100 con 0x7f05ec0c6a90 2023-12-04T05:40:27.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.710+0000 7f05f1bbe640 1 -- 172.21.15.107:0/3522863980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05ec0c81a0 con 0x7f05ec0c6a90 2023-12-04T05:40:27.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.710+0000 7f05e3fff640 1 -- 172.21.15.107:0/3522863980 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f05ec0c81a0 con 0x7f05ec0c6a90 2023-12-04T05:40:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:27.715 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":21,"fsid":"603e856b-31ac-475a-8f35-21ab7faadd48","created":"2023-12-04T05:39:34.921690+0000","modified":"2023-12-04T05:40:00.671650+0000","last_up_change":"2023-12-04T05:39:46.235790+0000","last_in_change":"2023-12-04T05:39:37.587639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T05:39:47.567702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T05:39:57.644286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T05:39:58.029897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ab7111c-0ae8-4d41-b6db-e3743614322c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6817","nonce":768714279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6821","nonce":768714279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6829","nonce":768714279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6825","nonce":768714279}]},"public_addr":"172.21.15.107:6817/768714279","cluster_addr":"172.21.15.107:6821/768714279","heartbeat_back_addr":"172.21.15.107:6829/768714279","heartbeat_front_addr":"172.21.15.107:6825/768714279","state":["exists","up"]},{"osd":1,"uuid":"baef7a1f-8026-4fab-9e70-48465271034e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6801","nonce":2496355981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6803","nonce":2496355981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6807","nonce":2496355981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6805","nonce":2496355981}]},"public_addr":"172.21.15.107:6801/2496355981","cluster_addr":"172.21.15.107:6803/2496355981","heartbeat_back_addr":"172.21.15.107:6807/2496355981","heartbeat_front_addr":"172.21.15.107:6805/2496355981","state":["exists","up"]},{"osd":2,"uuid":"90e8cf24-7b54-484a-9d85-b44a9d3dfeab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6809","nonce":3097298224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6811","nonce":3097298224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6815","nonce":3097298224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6813","nonce":3097298224}]},"public_addr":"172.21.15.107:6809/3097298224","cluster_addr":"172.21.15.107:6811/3097298224","heartbeat_back_addr":"172.21.15.107:6815/3097298224","heartbeat_front_addr":"172.21.15.107:6813/3097298224","state":["exists","up"]},{"osd":3,"uuid":"54658731-2b9a-4d5e-ab13-7c1be7d34e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6820","nonce":3773076671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6824","nonce":3773076671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6831","nonce":3773076671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6828","nonce":3773076671}]},"public_addr":"172.21.15.107:6820/3773076671","cluster_addr":"172.21.15.107:6824/3773076671","heartbeat_back_addr":"172.21.15.107:6831/3773076671","heartbeat_front_addr":"172.21.15.107:6828/3773076671","state":["exists","up"]},{"osd":4,"uuid":"9a9dea13-8afe-485d-aa43-45eda691179b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6801","nonce":2747823542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6803","nonce":2747823542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6807","nonce":2747823542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6805","nonce":2747823542}]},"public_addr":"172.21.15.195:6801/2747823542","cluster_addr":"172.21.15.195:6803/2747823542","heartbeat_back_addr":"172.21.15.195:6807/2747823542","heartbeat_front_addr":"172.21.15.195:6805/2747823542","state":["exists","up"]},{"osd":5,"uuid":"db840d83-5a7d-468a-8aa2-1997945847c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6825","nonce":1484264540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6827","nonce":1484264540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6831","nonce":1484264540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6829","nonce":1484264540}]},"public_addr":"172.21.15.195:6825/1484264540","cluster_addr":"172.21.15.195:6827/1484264540","heartbeat_back_addr":"172.21.15.195:6831/1484264540","heartbeat_front_addr":"172.21.15.195:6829/1484264540","state":["exists","up"]},{"osd":6,"uuid":"38b5f165-bf91-467e-a615-4b4784337689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6811","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6813","nonce":2638471412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6817","nonce":2638471412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6823","nonce":2638471412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6821","nonce":2638471412}]},"public_addr":"172.21.15.195:6813/2638471412","cluster_addr":"172.21.15.195:6817/2638471412","heartbeat_back_addr":"172.21.15.195:6823/2638471412","heartbeat_front_addr":"172.21.15.195:6821/2638471412","state":["exists","up"]},{"osd":7,"uuid":"e5b3c596-494b-4df3-9437-bc2d52168ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6809","nonce":768189614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6812","nonce":768189614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6820","nonce":768189614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6816","nonce":768189614}]},"public_addr":"172.21.15.195:6809/768189614","cluster_addr":"172.21.15.195:6812/768189614","heartbeat_back_addr":"172.21.15.195:6820/768189614","heartbeat_front_addr":"172.21.15.195:6816/768189614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.209077+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.086601+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.221568+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.238524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.059477+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.148422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.079867+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.064916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T05:40:27.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.714+0000 7f05f1bbe640 1 -- 172.21.15.107:0/3522863980 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f05cc0668d0 msgr2=0x7f05cc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:27.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.714+0000 7f05f1bbe640 1 --2- 172.21.15.107:0/3522863980 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f05cc0668d0 0x7f05cc068d90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f05dc000f90 tx=0x7f05dc009040 comp rx=0 tx=0).stop 2023-12-04T05:40:27.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.714+0000 7f05f1bbe640 1 -- 172.21.15.107:0/3522863980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05ec0c6a90 msgr2=0x7f05ec1524e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:27.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.714+0000 7f05f1bbe640 1 --2- 172.21.15.107:0/3522863980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05ec0c6a90 0x7f05ec1524e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f05d401c610 tx=0x7f05d8005800 comp rx=0 tx=0).stop 2023-12-04T05:40:27.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.714+0000 7f05f1bbe640 1 -- 172.21.15.107:0/3522863980 shutdown_connections 2023-12-04T05:40:27.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.714+0000 7f05f1bbe640 1 --2- 172.21.15.107:0/3522863980 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f05cc0668d0 0x7f05cc068d90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.714+0000 7f05f1bbe640 1 --2- 172.21.15.107:0/3522863980 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f05ec15cbf0 0x7f05ec149f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.714+0000 7f05f1bbe640 1 --2- 172.21.15.107:0/3522863980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05ec0c6a90 0x7f05ec1524e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.714+0000 7f05f1bbe640 1 --2- 172.21.15.107:0/3522863980 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f05d80047d0 0x7f05ec0b3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.714+0000 7f05f1bbe640 1 -- 172.21.15.107:0/3522863980 >> 172.21.15.107:0/3522863980 conn(0x7f05ec06c6c0 msgr2=0x7f05ec06a5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:27.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.714+0000 7f05f1bbe640 1 -- 172.21.15.107:0/3522863980 shutdown_connections 2023-12-04T05:40:27.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.714+0000 7f05f1bbe640 1 -- 172.21.15.107:0/3522863980 wait complete. 2023-12-04T05:40:27.733 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-04T05:39:47.567702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-04T05:39:57.644286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-04T05:39:58.029897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-04T05:40:27.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-04T05:40:27.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.838+0000 7f6f5c662640 1 Processor -- start 2023-12-04T05:40:27.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.838+0000 7f6f5c662640 1 -- start start 2023-12-04T05:40:27.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f5c662640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6f540c46a0 0x7f6f540c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f5c662640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f5406e660 con 0x7f6f540c4a80 2023-12-04T05:40:27.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f5c662640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f54156cf0 con 0x7f6f54157180 2023-12-04T05:40:27.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f5c662640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f54156490 con 0x7f6f540c46a0 2023-12-04T05:40:27.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f5abff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6f540c4a80 0x7f6f540c50e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42318/0 (socket says 172.21.15.107:42318) 2023-12-04T05:40:27.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f5abff640 1 -- 172.21.15.107:0/3995588192 learned_addr learned my addr 172.21.15.107:0/3995588192 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:27.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f5a3fe640 1 --2- 172.21.15.107:0/3995588192 >> v2:172.21.15.195:3300/0 conn(0x7f6f540c46a0 0x7f6f540c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:27.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f593fc640 1 -- 172.21.15.107:0/3995588192 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929537542 0 0) 0x7f6f4c0014a0 con 0x7f6f540c4a80 2023-12-04T05:40:27.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f593fc640 1 -- 172.21.15.107:0/3995588192 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f40003530 con 0x7f6f540c4a80 2023-12-04T05:40:27.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f593fc640 1 -- 172.21.15.107:0/3995588192 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3419775910 0 0) 0x7f6f40003530 con 0x7f6f540c4a80 2023-12-04T05:40:27.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f593fc640 1 -- 172.21.15.107:0/3995588192 >> v2:172.21.15.195:3300/0 conn(0x7f6f540c46a0 msgr2=0x7f6f540c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:27.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f593fc640 1 --2- 172.21.15.107:0/3995588192 >> v2:172.21.15.195:3300/0 conn(0x7f6f540c46a0 0x7f6f540c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f593fc640 1 -- 172.21.15.107:0/3995588192 >> v1:172.21.15.195:6789/0 conn(0x7f6f54157180 legacy=0x7f6f540c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:27.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f593fc640 1 -- 172.21.15.107:0/3995588192 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5406e0a0 con 0x7f6f540c4a80 2023-12-04T05:40:27.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f593fc640 1 -- 172.21.15.107:0/3995588192 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f6f4c001b90 con 0x7f6f540c4a80 2023-12-04T05:40:27.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f593fc640 1 -- 172.21.15.107:0/3995588192 >> v1:172.21.15.107:6789/0 conn(0x7f6f540c4a80 legacy=0x7f6f540c50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:27.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f593fc640 1 --2- 172.21.15.107:0/3995588192 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f40003980 0x7f6f40003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f593fc640 1 --2- 172.21.15.107:0/3995588192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f400047d0 0x7f6f40004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f593fc640 1 --2- 172.21.15.107:0/3995588192 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f40005190 0x7f6f4000caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f593fc640 1 -- 172.21.15.107:0/3995588192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f54156490 con 0x7f6f400047d0 2023-12-04T05:40:27.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f593fc640 1 -- 172.21.15.107:0/3995588192 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4000d140 con 0x7f6f40003980 2023-12-04T05:40:27.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f593fc640 1 -- 172.21.15.107:0/3995588192 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4000d300 con 0x7f6f40005190 2023-12-04T05:40:27.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f59bfd640 1 --2- 172.21.15.107:0/3995588192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f400047d0 0x7f6f40004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:27.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f5abff640 1 --2- 172.21.15.107:0/3995588192 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f40005190 0x7f6f4000caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:27.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f5a3fe640 1 --2- 172.21.15.107:0/3995588192 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f40003980 0x7f6f40003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:27.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f59bfd640 1 -- 172.21.15.107:0/3995588192 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f40005190 msgr2=0x7f6f4000caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:27.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f59bfd640 1 --2- 172.21.15.107:0/3995588192 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f40005190 0x7f6f4000caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f59bfd640 1 -- 172.21.15.107:0/3995588192 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f40003980 msgr2=0x7f6f40003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:27.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f59bfd640 1 --2- 172.21.15.107:0/3995588192 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f40003980 0x7f6f40003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f59bfd640 1 -- 172.21.15.107:0/3995588192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f40003530 con 0x7f6f400047d0 2023-12-04T05:40:27.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f5abff640 1 --2- 172.21.15.107:0/3995588192 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f40005190 0x7f6f4000caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:27.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f5a3fe640 1 --2- 172.21.15.107:0/3995588192 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f40003980 0x7f6f40003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:27.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f5abff640 1 -- 172.21.15.107:0/3995588192 reap_dead start 2023-12-04T05:40:27.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.842+0000 7f6f59bfd640 1 --2- 172.21.15.107:0/3995588192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f400047d0 0x7f6f40004c50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6f44001780 tx=0x7f6f4401c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1531e8b4aab7d4 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:27.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f593fc640 1 -- 172.21.15.107:0/3995588192 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f440052d0 con 0x7f6f400047d0 2023-12-04T05:40:27.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f593fc640 1 -- 172.21.15.107:0/3995588192 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f440a7050 con 0x7f6f400047d0 2023-12-04T05:40:27.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f593fc640 1 -- 172.21.15.107:0/3995588192 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f44003600 con 0x7f6f400047d0 2023-12-04T05:40:27.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5c662640 1 -- 172.21.15.107:0/3995588192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f400047d0 msgr2=0x7f6f40004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:27.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5c662640 1 --2- 172.21.15.107:0/3995588192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f400047d0 0x7f6f40004c50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6f44001780 tx=0x7f6f4401c390 comp rx=0 tx=0).stop 2023-12-04T05:40:27.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5c662640 1 -- 172.21.15.107:0/3995588192 shutdown_connections 2023-12-04T05:40:27.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5c662640 1 --2- 172.21.15.107:0/3995588192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f400047d0 0x7f6f40004c50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5c662640 1 -- 172.21.15.107:0/3995588192 >> 172.21.15.107:0/3995588192 conn(0x7f6f540c7ff0 msgr2=0x7f6f540c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:27.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5c662640 1 -- 172.21.15.107:0/3995588192 shutdown_connections 2023-12-04T05:40:27.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5c662640 1 -- 172.21.15.107:0/3995588192 wait complete. 2023-12-04T05:40:27.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5c662640 1 Processor -- start 2023-12-04T05:40:27.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5c662640 1 -- start start 2023-12-04T05:40:27.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5c662640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f400047d0 0x7f6f540c6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5c662640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f5414c8e0 0x7f6f54145ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5c662640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f54145ff0 0x7f6f541eb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5c662640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5414d7e0 con 0x7f6f54145ff0 2023-12-04T05:40:27.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5c662640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f540cb360 con 0x7f6f400047d0 2023-12-04T05:40:27.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5c662640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f541ef9c0 con 0x7f6f5414c8e0 2023-12-04T05:40:27.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5abff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f54145ff0 0x7f6f541eb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:27.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5abff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f54145ff0 0x7f6f541eb8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56292/0 (socket says 172.21.15.107:56292) 2023-12-04T05:40:27.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5abff640 1 -- 172.21.15.107:0/1805497846 learned_addr learned my addr 172.21.15.107:0/1805497846 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:27.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5a3fe640 1 --2- 172.21.15.107:0/1805497846 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f400047d0 0x7f6f540c6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:27.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f59bfd640 1 --2- 172.21.15.107:0/1805497846 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f5414c8e0 0x7f6f54145ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:27.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5abff640 1 -- 172.21.15.107:0/1805497846 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f5414c8e0 msgr2=0x7f6f54145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:27.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5abff640 1 --2- 172.21.15.107:0/1805497846 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f5414c8e0 0x7f6f54145ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5abff640 1 -- 172.21.15.107:0/1805497846 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f400047d0 msgr2=0x7f6f540c6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:27.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5abff640 1 --2- 172.21.15.107:0/1805497846 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f400047d0 0x7f6f540c6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:27.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5abff640 1 -- 172.21.15.107:0/1805497846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f541eead0 con 0x7f6f54145ff0 2023-12-04T05:40:27.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f59bfd640 1 --2- 172.21.15.107:0/1805497846 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f5414c8e0 0x7f6f54145ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:40:27.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5abff640 1 --2- 172.21.15.107:0/1805497846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f54145ff0 0x7f6f541eb8b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c004c00 tx=0x7f6f4c010bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:27.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f4affd640 1 -- 172.21.15.107:0/1805497846 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c00e8b0 con 0x7f6f54145ff0 2023-12-04T05:40:27.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f4affd640 1 -- 172.21.15.107:0/1805497846 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4c00ea10 con 0x7f6f54145ff0 2023-12-04T05:40:27.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5c662640 1 -- 172.21.15.107:0/1805497846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f541eedb0 con 0x7f6f54145ff0 2023-12-04T05:40:27.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f4affd640 1 -- 172.21.15.107:0/1805497846 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c00eda0 con 0x7f6f54145ff0 2023-12-04T05:40:27.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.846+0000 7f6f5c662640 1 -- 172.21.15.107:0/1805497846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f541ef370 con 0x7f6f54145ff0 2023-12-04T05:40:27.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.850+0000 7f6f5c662640 1 -- 172.21.15.107:0/1805497846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f541eb590 con 0x7f6f54145ff0 2023-12-04T05:40:27.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.850+0000 7f6f4affd640 1 -- 172.21.15.107:0/1805497846 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f6f4c0143a0 con 0x7f6f54145ff0 2023-12-04T05:40:27.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.850+0000 7f6f4affd640 1 --2- 172.21.15.107:0/1805497846 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6f38066900 0x7f6f38068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:27.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.850+0000 7f6f5a3fe640 1 --2- 172.21.15.107:0/1805497846 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6f38066900 0x7f6f38068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:27.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.850+0000 7f6f4affd640 1 -- 172.21.15.107:0/1805497846 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f4c086570 con 0x7f6f54145ff0 2023-12-04T05:40:27.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.850+0000 7f6f5a3fe640 1 --2- 172.21.15.107:0/1805497846 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6f38066900 0x7f6f38068dc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6f50009bf0 tx=0x7f6f50008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:27.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:27.854+0000 7f6f4affd640 1 -- 172.21.15.107:0/1805497846 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6f4c053850 con 0x7f6f54145ff0 2023-12-04T05:40:28.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.002+0000 7f6f5c662640 1 -- 172.21.15.107:0/1805497846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6f5406e660 con 0x7f6f54145ff0 2023-12-04T05:40:28.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.002+0000 7f6f4affd640 1 -- 172.21.15.107:0/1805497846 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6f5406e660 con 0x7f6f54145ff0 2023-12-04T05:40:28.005 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-12-04T05:40:28.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.002+0000 7f6f5c662640 1 -- 172.21.15.107:0/1805497846 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6f38066900 msgr2=0x7f6f38068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:28.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.002+0000 7f6f5c662640 1 --2- 172.21.15.107:0/1805497846 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6f38066900 0x7f6f38068dc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6f50009bf0 tx=0x7f6f50008040 comp rx=0 tx=0).stop 2023-12-04T05:40:28.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.002+0000 7f6f5c662640 1 -- 172.21.15.107:0/1805497846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f54145ff0 msgr2=0x7f6f541eb8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:28.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.002+0000 7f6f5c662640 1 --2- 172.21.15.107:0/1805497846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f54145ff0 0x7f6f541eb8b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c004c00 tx=0x7f6f4c010bb0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.002+0000 7f6f5c662640 1 -- 172.21.15.107:0/1805497846 shutdown_connections 2023-12-04T05:40:28.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.002+0000 7f6f5c662640 1 --2- 172.21.15.107:0/1805497846 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6f38066900 0x7f6f38068dc0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.002+0000 7f6f5c662640 1 --2- 172.21.15.107:0/1805497846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f54145ff0 0x7f6f541eb8b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.002+0000 7f6f5c662640 1 --2- 172.21.15.107:0/1805497846 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6f5414c8e0 0x7f6f54145ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.006+0000 7f6f5c662640 1 --2- 172.21.15.107:0/1805497846 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f400047d0 0x7f6f540c6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.006+0000 7f6f5c662640 1 -- 172.21.15.107:0/1805497846 >> 172.21.15.107:0/1805497846 conn(0x7f6f540c7ff0 msgr2=0x7f6f5414b510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:28.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.006+0000 7f6f5c662640 1 -- 172.21.15.107:0/1805497846 shutdown_connections 2023-12-04T05:40:28.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.006+0000 7f6f5c662640 1 -- 172.21.15.107:0/1805497846 wait complete. 2023-12-04T05:40:28.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-04T05:40:28.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.138+0000 7f1ef8b96640 1 Processor -- start 2023-12-04T05:40:28.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.138+0000 7f1ef8b96640 1 -- start start 2023-12-04T05:40:28.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef8b96640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1ef4071c00 0x7f1ef40c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:28.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef8b96640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ef40700c0 con 0x7f1ef4070380 2023-12-04T05:40:28.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef8b96640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ef4156df0 con 0x7f1ef4151180 2023-12-04T05:40:28.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef8b96640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef41570d0 con 0x7f1ef4071c00 2023-12-04T05:40:28.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef2d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1ef4070380 0x7f1ef4070790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42320/0 (socket says 172.21.15.107:42320) 2023-12-04T05:40:28.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef2d9d640 1 -- 172.21.15.107:0/592535159 learned_addr learned my addr 172.21.15.107:0/592535159 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:28.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ee9d9b640 1 --2- 172.21.15.107:0/592535159 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1ef4071c00 0x7f1ef40c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:28.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef1d9b640 1 -- 172.21.15.107:0/592535159 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222788113 0 0) 0x7f1ef40700c0 con 0x7f1ef4070380 2023-12-04T05:40:28.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef1d9b640 1 -- 172.21.15.107:0/592535159 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ee0003530 con 0x7f1ef4070380 2023-12-04T05:40:28.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef1d9b640 1 -- 172.21.15.107:0/592535159 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 610332712 0 0) 0x7f1ee0003530 con 0x7f1ef4070380 2023-12-04T05:40:28.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef1d9b640 1 -- 172.21.15.107:0/592535159 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1ef4071c00 msgr2=0x7f1ef40c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:28.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef1d9b640 1 --2- 172.21.15.107:0/592535159 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1ef4071c00 0x7f1ef40c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef1d9b640 1 -- 172.21.15.107:0/592535159 >> v1:172.21.15.195:6789/0 conn(0x7f1ef4151180 legacy=0x7f1ef40714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:28.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef1d9b640 1 -- 172.21.15.107:0/592535159 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef41573b0 con 0x7f1ef4070380 2023-12-04T05:40:28.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef1d9b640 1 -- 172.21.15.107:0/592535159 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f1eec001a00 con 0x7f1ef4070380 2023-12-04T05:40:28.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef1d9b640 1 -- 172.21.15.107:0/592535159 >> v1:172.21.15.107:6789/0 conn(0x7f1ef4070380 legacy=0x7f1ef4070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:28.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef1d9b640 1 --2- 172.21.15.107:0/592535159 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1ef4071c00 0x7f1ee00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:28.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef1d9b640 1 --2- 172.21.15.107:0/592535159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ee0003f30 0x7f1ee00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:28.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef1d9b640 1 --2- 172.21.15.107:0/592535159 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ee0004d10 0x7f1ee0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:28.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef1d9b640 1 -- 172.21.15.107:0/592535159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef41570d0 con 0x7f1ee0003f30 2023-12-04T05:40:28.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef1d9b640 1 -- 172.21.15.107:0/592535159 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee000ba70 con 0x7f1ee0004d10 2023-12-04T05:40:28.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef1d9b640 1 -- 172.21.15.107:0/592535159 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee000bc30 con 0x7f1ef4071c00 2023-12-04T05:40:28.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ee9d9b640 1 --2- 172.21.15.107:0/592535159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ee0003f30 0x7f1ee00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:28.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef259c640 1 --2- 172.21.15.107:0/592535159 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1ef4071c00 0x7f1ee00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:28.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ef2d9d640 1 --2- 172.21.15.107:0/592535159 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ee0004d10 0x7f1ee0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:28.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ee9d9b640 1 -- 172.21.15.107:0/592535159 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1ef4071c00 msgr2=0x7f1ee00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:28.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ee9d9b640 1 --2- 172.21.15.107:0/592535159 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1ef4071c00 0x7f1ee00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ee9d9b640 1 -- 172.21.15.107:0/592535159 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ee0004d10 msgr2=0x7f1ee0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:28.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ee9d9b640 1 --2- 172.21.15.107:0/592535159 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ee0004d10 0x7f1ee0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.142+0000 7f1ee9d9b640 1 -- 172.21.15.107:0/592535159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee0003530 con 0x7f1ee0003f30 2023-12-04T05:40:28.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ee9d9b640 1 --2- 172.21.15.107:0/592535159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ee0003f30 0x7f1ee00047d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1edc009ae0 tx=0x7f1edc01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fc91004d351f52c server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:28.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef1d9b640 1 -- 172.21.15.107:0/592535159 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1edc0a5530 con 0x7f1ee0003f30 2023-12-04T05:40:28.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef1d9b640 1 -- 172.21.15.107:0/592535159 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1edc0a5690 con 0x7f1ee0003f30 2023-12-04T05:40:28.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef1d9b640 1 -- 172.21.15.107:0/592535159 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1edc0a5a40 con 0x7f1ee0003f30 2023-12-04T05:40:28.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef8b96640 1 -- 172.21.15.107:0/592535159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ee0003f30 msgr2=0x7f1ee00047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:28.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef8b96640 1 --2- 172.21.15.107:0/592535159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ee0003f30 0x7f1ee00047d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1edc009ae0 tx=0x7f1edc01ca50 comp rx=0 tx=0).stop 2023-12-04T05:40:28.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef2d9d640 1 -- 172.21.15.107:0/592535159 reap_dead start 2023-12-04T05:40:28.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef8b96640 1 -- 172.21.15.107:0/592535159 shutdown_connections 2023-12-04T05:40:28.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef8b96640 1 -- 172.21.15.107:0/592535159 >> 172.21.15.107:0/592535159 conn(0x7f1ef40c7ff0 msgr2=0x7f1ef40c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:28.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef8b96640 1 -- 172.21.15.107:0/592535159 shutdown_connections 2023-12-04T05:40:28.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef8b96640 1 -- 172.21.15.107:0/592535159 wait complete. 2023-12-04T05:40:28.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef8b96640 1 Processor -- start 2023-12-04T05:40:28.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef8b96640 1 -- start start 2023-12-04T05:40:28.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef8b96640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1ee0003f30 0x7f1ef40c19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef8b96640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ef414e240 0x7f1ef40c5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef8b96640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ef41efe80 0x7f1ef41f0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef8b96640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef41f4350 con 0x7f1ef41efe80 2023-12-04T05:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef8b96640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef41f3ec0 con 0x7f1ef414e240 2023-12-04T05:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef8b96640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef40c59f0 con 0x7f1ee0003f30 2023-12-04T05:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ef41efe80 0x7f1ef41f0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ef41efe80 0x7f1ef41f0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56298/0 (socket says 172.21.15.107:56298) 2023-12-04T05:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef2d9d640 1 -- 172.21.15.107:0/2468759519 learned_addr learned my addr 172.21.15.107:0/2468759519 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef259c640 1 --2- 172.21.15.107:0/2468759519 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1ee0003f30 0x7f1ef40c19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ee9d9b640 1 --2- 172.21.15.107:0/2468759519 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ef414e240 0x7f1ef40c5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef2d9d640 1 -- 172.21.15.107:0/2468759519 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1ee0003f30 msgr2=0x7f1ef40c19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef2d9d640 1 --2- 172.21.15.107:0/2468759519 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1ee0003f30 0x7f1ef40c19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef2d9d640 1 -- 172.21.15.107:0/2468759519 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ef414e240 msgr2=0x7f1ef40c5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef2d9d640 1 --2- 172.21.15.107:0/2468759519 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ef414e240 0x7f1ef40c5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef2d9d640 1 -- 172.21.15.107:0/2468759519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef41f3590 con 0x7f1ef41efe80 2023-12-04T05:40:28.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef2d9d640 1 --2- 172.21.15.107:0/2468759519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ef41efe80 0x7f1ef41f0300 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1ef40c7fc0 tx=0x7f1eec011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:28.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1eeb7fe640 1 -- 172.21.15.107:0/2468759519 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eec00f920 con 0x7f1ef41efe80 2023-12-04T05:40:28.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1eeb7fe640 1 -- 172.21.15.107:0/2468759519 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eec00faa0 con 0x7f1ef41efe80 2023-12-04T05:40:28.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1eeb7fe640 1 -- 172.21.15.107:0/2468759519 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eec00fe50 con 0x7f1ef41efe80 2023-12-04T05:40:28.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef8b96640 1 -- 172.21.15.107:0/2468759519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ef41f3870 con 0x7f1ef41efe80 2023-12-04T05:40:28.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.146+0000 7f1ef8b96640 1 -- 172.21.15.107:0/2468759519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ef40c1f00 con 0x7f1ef41efe80 2023-12-04T05:40:28.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.150+0000 7f1ef8b96640 1 -- 172.21.15.107:0/2468759519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ef40c7d70 con 0x7f1ef41efe80 2023-12-04T05:40:28.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.150+0000 7f1eeb7fe640 1 -- 172.21.15.107:0/2468759519 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f1eec01f460 con 0x7f1ef41efe80 2023-12-04T05:40:28.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.150+0000 7f1eeb7fe640 1 --2- 172.21.15.107:0/2468759519 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1ed4066900 0x7f1ed4068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:28.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.150+0000 7f1ef259c640 1 --2- 172.21.15.107:0/2468759519 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1ed4066900 0x7f1ed4068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:28.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.150+0000 7f1eeb7fe640 1 -- 172.21.15.107:0/2468759519 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1eec087520 con 0x7f1ef41efe80 2023-12-04T05:40:28.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.150+0000 7f1ef259c640 1 --2- 172.21.15.107:0/2468759519 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1ed4066900 0x7f1ed4068dc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4005510 tx=0x7f1ee4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:28.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.154+0000 7f1eeb7fe640 1 -- 172.21.15.107:0/2468759519 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1eec0581f0 con 0x7f1ef41efe80 2023-12-04T05:40:28.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.310+0000 7f1ef8b96640 1 -- 172.21.15.107:0/2468759519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1ef41f4030 con 0x7f1ef41efe80 2023-12-04T05:40:28.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.310+0000 7f1eeb7fe640 1 -- 172.21.15.107:0/2468759519 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1eec057d80 con 0x7f1ef41efe80 2023-12-04T05:40:28.314 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-12-04T05:40:28.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.314+0000 7f1ef8b96640 1 -- 172.21.15.107:0/2468759519 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1ed4066900 msgr2=0x7f1ed4068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:28.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.314+0000 7f1ef8b96640 1 --2- 172.21.15.107:0/2468759519 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1ed4066900 0x7f1ed4068dc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4005510 tx=0x7f1ee4009040 comp rx=0 tx=0).stop 2023-12-04T05:40:28.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.314+0000 7f1ef8b96640 1 -- 172.21.15.107:0/2468759519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ef41efe80 msgr2=0x7f1ef41f0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:28.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.314+0000 7f1ef8b96640 1 --2- 172.21.15.107:0/2468759519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ef41efe80 0x7f1ef41f0300 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1ef40c7fc0 tx=0x7f1eec011c20 comp rx=0 tx=0).stop 2023-12-04T05:40:28.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.314+0000 7f1ef8b96640 1 -- 172.21.15.107:0/2468759519 shutdown_connections 2023-12-04T05:40:28.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.314+0000 7f1ef8b96640 1 --2- 172.21.15.107:0/2468759519 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1ed4066900 0x7f1ed4068dc0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.314+0000 7f1ef8b96640 1 --2- 172.21.15.107:0/2468759519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ef41efe80 0x7f1ef41f0300 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.314+0000 7f1ef8b96640 1 --2- 172.21.15.107:0/2468759519 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ef414e240 0x7f1ef40c5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.314+0000 7f1ef8b96640 1 --2- 172.21.15.107:0/2468759519 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1ee0003f30 0x7f1ef40c19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.314+0000 7f1ef8b96640 1 -- 172.21.15.107:0/2468759519 >> 172.21.15.107:0/2468759519 conn(0x7f1ef40c7ff0 msgr2=0x7f1ef4150160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.314+0000 7f1ef8b96640 1 -- 172.21.15.107:0/2468759519 shutdown_connections 2023-12-04T05:40:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.314+0000 7f1ef8b96640 1 -- 172.21.15.107:0/2468759519 wait complete. 2023-12-04T05:40:28.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-04T05:40:28.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.450+0000 7f86d5f3e640 1 Processor -- start 2023-12-04T05:40:28.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.450+0000 7f86d5f3e640 1 -- start start 2023-12-04T05:40:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.450+0000 7f86d5f3e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f86d00c56f0 0x7f86d0158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.450+0000 7f86d5f3e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d0158da0 con 0x7f86d0072110 2023-12-04T05:40:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.450+0000 7f86d5f3e640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d01583d0 con 0x7f86d00724f0 2023-12-04T05:40:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.450+0000 7f86d5f3e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86d00c6170 con 0x7f86d00c56f0 2023-12-04T05:40:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.450+0000 7f86cffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f86d00c56f0 0x7f86d0158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.450+0000 7f86cf7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f86d0072110 0x7f86d00c7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42336/0 (socket says 172.21.15.107:42336) 2023-12-04T05:40:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.450+0000 7f86cffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f86d00c56f0 0x7f86d0158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56312/0 (socket says 172.21.15.107:56312) 2023-12-04T05:40:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.450+0000 7f86cf7fe640 1 -- 172.21.15.107:0/1037595548 learned_addr learned my addr 172.21.15.107:0/1037595548 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:28.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.450+0000 7f86cffff640 1 -- 172.21.15.107:0/1037595548 >> v1:172.21.15.195:6789/0 conn(0x7f86d00724f0 legacy=0x7f86d0140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:28.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.450+0000 7f86cffff640 1 -- 172.21.15.107:0/1037595548 >> v1:172.21.15.107:6789/0 conn(0x7f86d0072110 legacy=0x7f86d00c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:28.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.450+0000 7f86cffff640 1 -- 172.21.15.107:0/1037595548 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d0157da0 con 0x7f86d00c56f0 2023-12-04T05:40:28.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86cffff640 1 --2- 172.21.15.107:0/1037595548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d00c56f0 0x7f86d0158860 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f86c8009d90 tx=0x7f86c801f100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fe25da46d5c9e2f server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:28.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86ce7fc640 1 -- 172.21.15.107:0/1037595548 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c80af600 con 0x7f86d00c56f0 2023-12-04T05:40:28.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86ce7fc640 1 -- 172.21.15.107:0/1037595548 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c80ab050 con 0x7f86d00c56f0 2023-12-04T05:40:28.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86ce7fc640 1 -- 172.21.15.107:0/1037595548 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c80af980 con 0x7f86d00c56f0 2023-12-04T05:40:28.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86d5f3e640 1 -- 172.21.15.107:0/1037595548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d00c56f0 msgr2=0x7f86d0158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:28.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86d5f3e640 1 --2- 172.21.15.107:0/1037595548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d00c56f0 0x7f86d0158860 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f86c8009d90 tx=0x7f86c801f100 comp rx=0 tx=0).stop 2023-12-04T05:40:28.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86d5f3e640 1 -- 172.21.15.107:0/1037595548 shutdown_connections 2023-12-04T05:40:28.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86d5f3e640 1 --2- 172.21.15.107:0/1037595548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d00c56f0 0x7f86d0158860 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86d5f3e640 1 -- 172.21.15.107:0/1037595548 >> 172.21.15.107:0/1037595548 conn(0x7f86d00c7ff0 msgr2=0x7f86d00c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86d5f3e640 1 -- 172.21.15.107:0/1037595548 shutdown_connections 2023-12-04T05:40:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86d5f3e640 1 -- 172.21.15.107:0/1037595548 wait complete. 2023-12-04T05:40:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86d5f3e640 1 Processor -- start 2023-12-04T05:40:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86d5f3e640 1 -- start start 2023-12-04T05:40:28.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86d5f3e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d0072110 0x7f86d0070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86d5f3e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86d00724f0 0x7f86d0153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86d5f3e640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86d00c56f0 0x7f86d014cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86d5f3e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d01f2ab0 con 0x7f86d0072110 2023-12-04T05:40:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86d5f3e640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d0070910 con 0x7f86d00c56f0 2023-12-04T05:40:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86d5f3e640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86d0070570 con 0x7f86d00724f0 2023-12-04T05:40:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86cf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d0072110 0x7f86d0070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86cf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d0072110 0x7f86d0070030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56318/0 (socket says 172.21.15.107:56318) 2023-12-04T05:40:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86cf7fe640 1 -- 172.21.15.107:0/3910653906 learned_addr learned my addr 172.21.15.107:0/3910653906 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86ceffd640 1 --2- 172.21.15.107:0/3910653906 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86d00724f0 0x7f86d0153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86cffff640 1 --2- 172.21.15.107:0/3910653906 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86d00c56f0 0x7f86d014cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86cf7fe640 1 -- 172.21.15.107:0/3910653906 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86d00724f0 msgr2=0x7f86d0153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86cf7fe640 1 --2- 172.21.15.107:0/3910653906 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86d00724f0 0x7f86d0153030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86cf7fe640 1 -- 172.21.15.107:0/3910653906 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86d00c56f0 msgr2=0x7f86d014cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86cf7fe640 1 --2- 172.21.15.107:0/3910653906 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86d00c56f0 0x7f86d014cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86cf7fe640 1 -- 172.21.15.107:0/3910653906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c8009a70 con 0x7f86d0072110 2023-12-04T05:40:28.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86ceffd640 1 --2- 172.21.15.107:0/3910653906 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86d00724f0 0x7f86d0153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:28.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86cffff640 1 --2- 172.21.15.107:0/3910653906 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86d00c56f0 0x7f86d014cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:28.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.454+0000 7f86cf7fe640 1 --2- 172.21.15.107:0/3910653906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d0072110 0x7f86d0070030 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f86c000c500 tx=0x7f86c000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:28.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.458+0000 7f86b7fff640 1 -- 172.21.15.107:0/3910653906 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c0005cf0 con 0x7f86d0072110 2023-12-04T05:40:28.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.458+0000 7f86b7fff640 1 -- 172.21.15.107:0/3910653906 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c0005e50 con 0x7f86d0072110 2023-12-04T05:40:28.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.458+0000 7f86b7fff640 1 -- 172.21.15.107:0/3910653906 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c00072f0 con 0x7f86d0072110 2023-12-04T05:40:28.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.458+0000 7f86d5f3e640 1 -- 172.21.15.107:0/3910653906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d015bf70 con 0x7f86d0072110 2023-12-04T05:40:28.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.458+0000 7f86d5f3e640 1 -- 172.21.15.107:0/3910653906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d0152ba0 con 0x7f86d0072110 2023-12-04T05:40:28.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.458+0000 7f86d5f3e640 1 -- 172.21.15.107:0/3910653906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d0067870 con 0x7f86d0072110 2023-12-04T05:40:28.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.458+0000 7f86b7fff640 1 -- 172.21.15.107:0/3910653906 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f86c0007450 con 0x7f86d0072110 2023-12-04T05:40:28.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.458+0000 7f86b7fff640 1 --2- 172.21.15.107:0/3910653906 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f86bc0668d0 0x7f86bc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:28.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.458+0000 7f86b7fff640 1 -- 172.21.15.107:0/3910653906 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86c0083740 con 0x7f86d0072110 2023-12-04T05:40:28.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.458+0000 7f86ceffd640 1 --2- 172.21.15.107:0/3910653906 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f86bc0668d0 0x7f86bc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:28.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.458+0000 7f86ceffd640 1 --2- 172.21.15.107:0/3910653906 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f86bc0668d0 0x7f86bc068d90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f86d01ecd40 tx=0x7f86c4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:28.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.462+0000 7f86b7fff640 1 -- 172.21.15.107:0/3910653906 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f86c0050ac0 con 0x7f86d0072110 2023-12-04T05:40:28.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.606+0000 7f86d5f3e640 1 -- 172.21.15.107:0/3910653906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f86d0158da0 con 0x7f86d0072110 2023-12-04T05:40:28.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.606+0000 7f86b7fff640 1 -- 172.21.15.107:0/3910653906 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f86d0158da0 con 0x7f86d0072110 2023-12-04T05:40:28.609 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-12-04T05:40:28.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.606+0000 7f86d5f3e640 1 -- 172.21.15.107:0/3910653906 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f86bc0668d0 msgr2=0x7f86bc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:28.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.606+0000 7f86d5f3e640 1 --2- 172.21.15.107:0/3910653906 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f86bc0668d0 0x7f86bc068d90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f86d01ecd40 tx=0x7f86c4009040 comp rx=0 tx=0).stop 2023-12-04T05:40:28.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.606+0000 7f86d5f3e640 1 -- 172.21.15.107:0/3910653906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d0072110 msgr2=0x7f86d0070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:28.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.606+0000 7f86d5f3e640 1 --2- 172.21.15.107:0/3910653906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d0072110 0x7f86d0070030 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f86c000c500 tx=0x7f86c000c9c0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.606+0000 7f86d5f3e640 1 -- 172.21.15.107:0/3910653906 shutdown_connections 2023-12-04T05:40:28.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.606+0000 7f86d5f3e640 1 --2- 172.21.15.107:0/3910653906 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f86bc0668d0 0x7f86bc068d90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.606+0000 7f86d5f3e640 1 --2- 172.21.15.107:0/3910653906 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86d00c56f0 0x7f86d014cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.606+0000 7f86d5f3e640 1 --2- 172.21.15.107:0/3910653906 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f86d00724f0 0x7f86d0153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.606+0000 7f86d5f3e640 1 --2- 172.21.15.107:0/3910653906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d0072110 0x7f86d0070030 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.606+0000 7f86d5f3e640 1 -- 172.21.15.107:0/3910653906 >> 172.21.15.107:0/3910653906 conn(0x7f86d00c7ff0 msgr2=0x7f86d00c6340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:28.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.606+0000 7f86d5f3e640 1 -- 172.21.15.107:0/3910653906 shutdown_connections 2023-12-04T05:40:28.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.606+0000 7f86d5f3e640 1 -- 172.21.15.107:0/3910653906 wait complete. 2023-12-04T05:40:28.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:40:28.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.742+0000 7f2c425f8640 1 Processor -- start 2023-12-04T05:40:28.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.742+0000 7f2c425f8640 1 -- start start 2023-12-04T05:40:28.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c425f8640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2c3c072110 0x7f2c3c158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:28.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c425f8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2c3c0724f0 0x7f2c3c0c7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:28.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c425f8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c3c158da0 con 0x7f2c3c0c56f0 2023-12-04T05:40:28.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c425f8640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c3c1583d0 con 0x7f2c3c0724f0 2023-12-04T05:40:28.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c425f8640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c0c6170 con 0x7f2c3c072110 2023-12-04T05:40:28.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c3b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2c3c0724f0 0x7f2c3c0c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:28.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c3b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2c3c0724f0 0x7f2c3c0c7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56330/0 (socket says 172.21.15.107:56330) 2023-12-04T05:40:28.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c3b7fe640 1 -- 172.21.15.107:0/3164199155 learned_addr learned my addr 172.21.15.107:0/3164199155 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:28.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c40b95640 1 --1- 172.21.15.107:0/3164199155 >> v1:172.21.15.107:6789/0 conn(0x7f2c3c0c56f0 0x7f2c3c140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42344/0 (socket says 172.21.15.107:42344) 2023-12-04T05:40:28.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c3bfff640 1 --2- 172.21.15.107:0/3164199155 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2c3c072110 0x7f2c3c158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:28.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c3b7fe640 1 -- 172.21.15.107:0/3164199155 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2c3c072110 msgr2=0x7f2c3c158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:28.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c3b7fe640 1 --2- 172.21.15.107:0/3164199155 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2c3c072110 0x7f2c3c158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c3b7fe640 1 -- 172.21.15.107:0/3164199155 >> v1:172.21.15.107:6789/0 conn(0x7f2c3c0c56f0 legacy=0x7f2c3c140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:28.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c3b7fe640 1 -- 172.21.15.107:0/3164199155 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3c157da0 con 0x7f2c3c0724f0 2023-12-04T05:40:28.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c3affd640 1 -- 172.21.15.107:0/3164199155 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2494620297 0 0) 0x7f2c2c001120 con 0x7f2c3c0c56f0 2023-12-04T05:40:28.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c3b7fe640 1 --2- 172.21.15.107:0/3164199155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c3c0724f0 0x7f2c3c0c7420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2c30009870 tx=0x7f2c3001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4703f2fe8c7dc882 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:28.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c3affd640 1 -- 172.21.15.107:0/3164199155 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c300a7070 con 0x7f2c3c0724f0 2023-12-04T05:40:28.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c3affd640 1 -- 172.21.15.107:0/3164199155 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c300ab430 con 0x7f2c3c0724f0 2023-12-04T05:40:28.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c3affd640 1 -- 172.21.15.107:0/3164199155 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c300ab7c0 con 0x7f2c3c0724f0 2023-12-04T05:40:28.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c425f8640 1 -- 172.21.15.107:0/3164199155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c3c0724f0 msgr2=0x7f2c3c0c7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:28.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c425f8640 1 --2- 172.21.15.107:0/3164199155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c3c0724f0 0x7f2c3c0c7420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2c30009870 tx=0x7f2c3001c540 comp rx=0 tx=0).stop 2023-12-04T05:40:28.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c425f8640 1 -- 172.21.15.107:0/3164199155 shutdown_connections 2023-12-04T05:40:28.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c425f8640 1 --2- 172.21.15.107:0/3164199155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c3c0724f0 0x7f2c3c0c7420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c425f8640 1 --2- 172.21.15.107:0/3164199155 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2c3c072110 0x7f2c3c158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c425f8640 1 -- 172.21.15.107:0/3164199155 >> 172.21.15.107:0/3164199155 conn(0x7f2c3c0c7ff0 msgr2=0x7f2c3c0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:28.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c425f8640 1 -- 172.21.15.107:0/3164199155 shutdown_connections 2023-12-04T05:40:28.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.746+0000 7f2c425f8640 1 -- 172.21.15.107:0/3164199155 wait complete. 2023-12-04T05:40:28.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c425f8640 1 Processor -- start 2023-12-04T05:40:28.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c425f8640 1 -- start start 2023-12-04T05:40:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c425f8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c3c072110 0x7f2c3c13f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c425f8640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2c3c0724f0 0x7f2c3c13cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c425f8640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2c3c0c56f0 0x7f2c3c13e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c425f8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c1f18c0 con 0x7f2c3c072110 2023-12-04T05:40:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c425f8640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c1f1320 con 0x7f2c3c0c56f0 2023-12-04T05:40:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c425f8640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c1f15e0 con 0x7f2c3c0724f0 2023-12-04T05:40:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c3bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c3c072110 0x7f2c3c13f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c3bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c3c072110 0x7f2c3c13f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56334/0 (socket says 172.21.15.107:56334) 2023-12-04T05:40:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c3bfff640 1 -- 172.21.15.107:0/3728484562 learned_addr learned my addr 172.21.15.107:0/3728484562 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c40b95640 1 --2- 172.21.15.107:0/3728484562 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2c3c0c56f0 0x7f2c3c13e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c3b7fe640 1 --2- 172.21.15.107:0/3728484562 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2c3c0724f0 0x7f2c3c13cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c3bfff640 1 -- 172.21.15.107:0/3728484562 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2c3c0724f0 msgr2=0x7f2c3c13cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c3bfff640 1 --2- 172.21.15.107:0/3728484562 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2c3c0724f0 0x7f2c3c13cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c3bfff640 1 -- 172.21.15.107:0/3728484562 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2c3c0c56f0 msgr2=0x7f2c3c13e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:28.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c3bfff640 1 --2- 172.21.15.107:0/3728484562 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2c3c0c56f0 0x7f2c3c13e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c3bfff640 1 -- 172.21.15.107:0/3728484562 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c300094d0 con 0x7f2c3c072110 2023-12-04T05:40:28.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c40b95640 1 --2- 172.21.15.107:0/3728484562 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2c3c0c56f0 0x7f2c3c13e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:28.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c3bfff640 1 --2- 172.21.15.107:0/3728484562 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c3c072110 0x7f2c3c13f850 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2c34007e40 tx=0x7f2c3400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:28.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c38ff9640 1 -- 172.21.15.107:0/3728484562 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c34014620 con 0x7f2c3c072110 2023-12-04T05:40:28.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c38ff9640 1 -- 172.21.15.107:0/3728484562 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3400ee90 con 0x7f2c3c072110 2023-12-04T05:40:28.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c38ff9640 1 -- 172.21.15.107:0/3728484562 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c340149c0 con 0x7f2c3c072110 2023-12-04T05:40:28.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c425f8640 1 -- 172.21.15.107:0/3728484562 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c3c140740 con 0x7f2c3c072110 2023-12-04T05:40:28.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c425f8640 1 -- 172.21.15.107:0/3728484562 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c3c140d00 con 0x7f2c3c072110 2023-12-04T05:40:28.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.750+0000 7f2c425f8640 1 -- 172.21.15.107:0/3728484562 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c3c140580 con 0x7f2c3c072110 2023-12-04T05:40:28.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.754+0000 7f2c38ff9640 1 -- 172.21.15.107:0/3728484562 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f2c34014b20 con 0x7f2c3c072110 2023-12-04T05:40:28.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.754+0000 7f2c38ff9640 1 --2- 172.21.15.107:0/3728484562 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2c280668d0 0x7f2c28068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:28.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.754+0000 7f2c3b7fe640 1 --2- 172.21.15.107:0/3728484562 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2c280668d0 0x7f2c28068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:28.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.754+0000 7f2c38ff9640 1 -- 172.21.15.107:0/3728484562 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c34084e30 con 0x7f2c3c072110 2023-12-04T05:40:28.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.754+0000 7f2c3b7fe640 1 --2- 172.21.15.107:0/3728484562 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2c280668d0 0x7f2c28068d90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2c300af000 tx=0x7f2c30008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:28.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.758+0000 7f2c38ff9640 1 -- 172.21.15.107:0/3728484562 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2c34051fe0 con 0x7f2c3c072110 2023-12-04T05:40:28.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.938+0000 7f2c425f8640 1 -- 172.21.15.107:0/3728484562 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c3c067870 con 0x7f2c3c072110 2023-12-04T05:40:28.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.938+0000 7f2c38ff9640 1 -- 172.21.15.107:0/3728484562 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+6564 (secure 0 0 0) 0x7f2c34055600 con 0x7f2c3c072110 2023-12-04T05:40:28.942 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:28.942 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:40:23.915736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":16,"state":"up:creating","state_seq":1,"addr":"172.21.15.195:6836/3769196693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":3769196693},{"type":"v1","addr":"172.21.15.195:6836","nonce":3769196693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:40:28.942 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 16 2023-12-04T05:40:28.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.938+0000 7f2c425f8640 1 -- 172.21.15.107:0/3728484562 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2c280668d0 msgr2=0x7f2c28068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:28.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.938+0000 7f2c425f8640 1 --2- 172.21.15.107:0/3728484562 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2c280668d0 0x7f2c28068d90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2c300af000 tx=0x7f2c30008040 comp rx=0 tx=0).stop 2023-12-04T05:40:28.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.938+0000 7f2c425f8640 1 -- 172.21.15.107:0/3728484562 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c3c072110 msgr2=0x7f2c3c13f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:28.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.938+0000 7f2c425f8640 1 --2- 172.21.15.107:0/3728484562 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c3c072110 0x7f2c3c13f850 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2c34007e40 tx=0x7f2c3400e4b0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.938+0000 7f2c425f8640 1 -- 172.21.15.107:0/3728484562 shutdown_connections 2023-12-04T05:40:28.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.938+0000 7f2c425f8640 1 --2- 172.21.15.107:0/3728484562 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2c280668d0 0x7f2c28068d90 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.938+0000 7f2c425f8640 1 --2- 172.21.15.107:0/3728484562 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2c3c0c56f0 0x7f2c3c13e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.938+0000 7f2c425f8640 1 --2- 172.21.15.107:0/3728484562 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2c3c0724f0 0x7f2c3c13cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.938+0000 7f2c425f8640 1 --2- 172.21.15.107:0/3728484562 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c3c072110 0x7f2c3c13f850 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:28.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.938+0000 7f2c425f8640 1 -- 172.21.15.107:0/3728484562 >> 172.21.15.107:0/3728484562 conn(0x7f2c3c0c7ff0 msgr2=0x7f2c3c1574f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.938+0000 7f2c425f8640 1 -- 172.21.15.107:0/3728484562 shutdown_connections 2023-12-04T05:40:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:28.942+0000 7f2c425f8640 1 -- 172.21.15.107:0/3728484562 wait complete. 2023-12-04T05:40:28.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T05:40:29.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bcbc8f640 1 Processor -- start 2023-12-04T05:40:29.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bcbc8f640 1 -- start start 2023-12-04T05:40:29.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bcbc8f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3bc40c4c10 0x7f3bc40c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:29.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bcbc8f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bc415d710 con 0x7f3bc40d01c0 2023-12-04T05:40:29.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bcbc8f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bc415d9f0 con 0x7f3bc4154020 2023-12-04T05:40:29.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bcbc8f640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bc40ca230 con 0x7f3bc40c4c10 2023-12-04T05:40:29.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc922a640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3bc40d01c0 0x7f3bc4071ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42352/0 (socket says 172.21.15.107:42352) 2023-12-04T05:40:29.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc922a640 1 -- 172.21.15.107:0/2153614009 learned_addr learned my addr 172.21.15.107:0/2153614009 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:29.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bca22c640 1 --2- 172.21.15.107:0/2153614009 >> v2:172.21.15.195:3300/0 conn(0x7f3bc40c4c10 0x7f3bc40c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:29.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc8a29640 1 -- 172.21.15.107:0/2153614009 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606019259 0 0) 0x7f3bc415d710 con 0x7f3bc40d01c0 2023-12-04T05:40:29.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc8a29640 1 -- 172.21.15.107:0/2153614009 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bb0003530 con 0x7f3bc40d01c0 2023-12-04T05:40:29.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc8a29640 1 -- 172.21.15.107:0/2153614009 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3170945232 0 0) 0x7f3bb0003530 con 0x7f3bc40d01c0 2023-12-04T05:40:29.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc8a29640 1 -- 172.21.15.107:0/2153614009 >> v2:172.21.15.195:3300/0 conn(0x7f3bc40c4c10 msgr2=0x7f3bc40c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:29.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc8a29640 1 --2- 172.21.15.107:0/2153614009 >> v2:172.21.15.195:3300/0 conn(0x7f3bc40c4c10 0x7f3bc40c5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:29.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc8a29640 1 -- 172.21.15.107:0/2153614009 >> v1:172.21.15.195:6789/0 conn(0x7f3bc4154020 legacy=0x7f3bc415a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:29.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc8a29640 1 -- 172.21.15.107:0/2153614009 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc415dcd0 con 0x7f3bc40d01c0 2023-12-04T05:40:29.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc8a29640 1 -- 172.21.15.107:0/2153614009 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f3bb4001680 con 0x7f3bc40d01c0 2023-12-04T05:40:29.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc8a29640 1 -- 172.21.15.107:0/2153614009 >> v1:172.21.15.107:6789/0 conn(0x7f3bc40d01c0 legacy=0x7f3bc4071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:29.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc8a29640 1 --2- 172.21.15.107:0/2153614009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb0003980 0x7f3bb0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:29.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc8a29640 1 --2- 172.21.15.107:0/2153614009 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3bb00047d0 0x7f3bb0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:29.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc8a29640 1 --2- 172.21.15.107:0/2153614009 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3bb0005190 0x7f3bb000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:29.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc8a29640 1 -- 172.21.15.107:0/2153614009 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb000c010 con 0x7f3bb0003980 2023-12-04T05:40:29.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc8a29640 1 -- 172.21.15.107:0/2153614009 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb000c180 con 0x7f3bb0005190 2023-12-04T05:40:29.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc8a29640 1 -- 172.21.15.107:0/2153614009 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb000c340 con 0x7f3bb00047d0 2023-12-04T05:40:29.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc9a2b640 1 --2- 172.21.15.107:0/2153614009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb0003980 0x7f3bb0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:29.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bca22c640 1 --2- 172.21.15.107:0/2153614009 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3bb0005190 0x7f3bb000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:29.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc922a640 1 --2- 172.21.15.107:0/2153614009 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3bb00047d0 0x7f3bb0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:29.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc9a2b640 1 -- 172.21.15.107:0/2153614009 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3bb00047d0 msgr2=0x7f3bb0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:29.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc9a2b640 1 --2- 172.21.15.107:0/2153614009 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3bb00047d0 0x7f3bb0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:29.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc9a2b640 1 -- 172.21.15.107:0/2153614009 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3bb0005190 msgr2=0x7f3bb000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:29.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc9a2b640 1 --2- 172.21.15.107:0/2153614009 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3bb0005190 0x7f3bb000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:29.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bca22c640 1 -- 172.21.15.107:0/2153614009 reap_dead start 2023-12-04T05:40:29.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.074+0000 7f3bc9a2b640 1 -- 172.21.15.107:0/2153614009 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb0003530 con 0x7f3bb0003980 2023-12-04T05:40:29.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bc9a2b640 1 --2- 172.21.15.107:0/2153614009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb0003980 0x7f3bb0003da0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0001780 tx=0x7f3bc001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2028b1d29ec2c2a2 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:29.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bc8a29640 1 -- 172.21.15.107:0/2153614009 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc00a4ea0 con 0x7f3bb0003980 2023-12-04T05:40:29.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bc8a29640 1 -- 172.21.15.107:0/2153614009 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc00a7050 con 0x7f3bb0003980 2023-12-04T05:40:29.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bc8a29640 1 -- 172.21.15.107:0/2153614009 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc0003600 con 0x7f3bb0003980 2023-12-04T05:40:29.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bcbc8f640 1 -- 172.21.15.107:0/2153614009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb0003980 msgr2=0x7f3bb0003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:29.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bcbc8f640 1 --2- 172.21.15.107:0/2153614009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb0003980 0x7f3bb0003da0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0001780 tx=0x7f3bc001c3e0 comp rx=0 tx=0).stop 2023-12-04T05:40:29.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bcbc8f640 1 -- 172.21.15.107:0/2153614009 shutdown_connections 2023-12-04T05:40:29.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bcbc8f640 1 --2- 172.21.15.107:0/2153614009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb0003980 0x7f3bb0003da0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:29.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bcbc8f640 1 -- 172.21.15.107:0/2153614009 >> 172.21.15.107:0/2153614009 conn(0x7f3bc406c6c0 msgr2=0x7f3bc40cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:29.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bcbc8f640 1 -- 172.21.15.107:0/2153614009 shutdown_connections 2023-12-04T05:40:29.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bcbc8f640 1 -- 172.21.15.107:0/2153614009 wait complete. 2023-12-04T05:40:29.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bcbc8f640 1 Processor -- start 2023-12-04T05:40:29.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bcbc8f640 1 -- start start 2023-12-04T05:40:29.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bcbc8f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3bb0003980 0x7f3bc415d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:29.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bcbc8f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bc40c9580 0x7f3bc40ce570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:29.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bcbc8f640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3bc40ceab0 0x7f3bc40c8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:29.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bcbc8f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc415c2b0 con 0x7f3bc40c9580 2023-12-04T05:40:29.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bcbc8f640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc415bfd0 con 0x7f3bc40ceab0 2023-12-04T05:40:29.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bcbc8f640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc41ef850 con 0x7f3bb0003980 2023-12-04T05:40:29.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bc922a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bc40c9580 0x7f3bc40ce570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:29.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bc922a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bc40c9580 0x7f3bc40ce570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56356/0 (socket says 172.21.15.107:56356) 2023-12-04T05:40:29.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bc922a640 1 -- 172.21.15.107:0/1764966617 learned_addr learned my addr 172.21.15.107:0/1764966617 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:29.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.078+0000 7f3bc9a2b640 1 --2- 172.21.15.107:0/1764966617 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3bb0003980 0x7f3bc415d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:29.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.082+0000 7f3bca22c640 1 --2- 172.21.15.107:0/1764966617 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3bc40ceab0 0x7f3bc40c8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:29.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.082+0000 7f3bc922a640 1 -- 172.21.15.107:0/1764966617 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3bb0003980 msgr2=0x7f3bc415d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:29.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.082+0000 7f3bc922a640 1 --2- 172.21.15.107:0/1764966617 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3bb0003980 0x7f3bc415d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:29.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.082+0000 7f3bc922a640 1 -- 172.21.15.107:0/1764966617 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3bc40ceab0 msgr2=0x7f3bc40c8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:29.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.082+0000 7f3bc922a640 1 --2- 172.21.15.107:0/1764966617 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3bc40ceab0 0x7f3bc40c8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:29.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.082+0000 7f3bc922a640 1 -- 172.21.15.107:0/1764966617 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc41efb30 con 0x7f3bc40c9580 2023-12-04T05:40:29.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.082+0000 7f3bc922a640 1 --2- 172.21.15.107:0/1764966617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bc40c9580 0x7f3bc40ce570 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40118d0 tx=0x7f3bb4011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:29.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.082+0000 7f3bba7fc640 1 -- 172.21.15.107:0/1764966617 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb400fb60 con 0x7f3bc40c9580 2023-12-04T05:40:29.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.082+0000 7f3bba7fc640 1 -- 172.21.15.107:0/1764966617 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb400fce0 con 0x7f3bc40c9580 2023-12-04T05:40:29.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.082+0000 7f3bba7fc640 1 -- 172.21.15.107:0/1764966617 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4015650 con 0x7f3bc40c9580 2023-12-04T05:40:29.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.082+0000 7f3bcbc8f640 1 -- 172.21.15.107:0/1764966617 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bc41efe10 con 0x7f3bc40c9580 2023-12-04T05:40:29.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.082+0000 7f3bcbc8f640 1 -- 172.21.15.107:0/1764966617 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bc41f0610 con 0x7f3bc40c9580 2023-12-04T05:40:29.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.082+0000 7f3bcbc8f640 1 -- 172.21.15.107:0/1764966617 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bc40c57c0 con 0x7f3bc40c9580 2023-12-04T05:40:29.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.082+0000 7f3bba7fc640 1 -- 172.21.15.107:0/1764966617 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f3bb40157b0 con 0x7f3bc40c9580 2023-12-04T05:40:29.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.082+0000 7f3bba7fc640 1 --2- 172.21.15.107:0/1764966617 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3ba80668d0 0x7f3ba8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:29.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.082+0000 7f3bc9a2b640 1 --2- 172.21.15.107:0/1764966617 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3ba80668d0 0x7f3ba8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:29.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.082+0000 7f3bba7fc640 1 -- 172.21.15.107:0/1764966617 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3bb4087790 con 0x7f3bc40c9580 2023-12-04T05:40:29.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.082+0000 7f3bc9a2b640 1 --2- 172.21.15.107:0/1764966617 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3ba80668d0 0x7f3ba8068d90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0001750 tx=0x7f3bc00011a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:29.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.086+0000 7f3bba7fc640 1 -- 172.21.15.107:0/1764966617 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3bb4054a40 con 0x7f3bc40c9580 2023-12-04T05:40:29.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.242+0000 7f3bcbc8f640 1 -- 172.21.15.107:0/1764966617 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bc4067870 con 0x7f3bc40c9580 2023-12-04T05:40:29.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.246+0000 7f3bba7fc640 1 -- 172.21.15.107:0/1764966617 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f3bb4058060 con 0x7f3bc40c9580 2023-12-04T05:40:29.249 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:29.249 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":21,"fsid":"603e856b-31ac-475a-8f35-21ab7faadd48","created":"2023-12-04T05:39:34.921690+0000","modified":"2023-12-04T05:40:00.671650+0000","last_up_change":"2023-12-04T05:39:46.235790+0000","last_in_change":"2023-12-04T05:39:37.587639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T05:39:47.567702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T05:39:57.644286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T05:39:58.029897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ab7111c-0ae8-4d41-b6db-e3743614322c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6817","nonce":768714279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6821","nonce":768714279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6829","nonce":768714279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6825","nonce":768714279}]},"public_addr":"172.21.15.107:6817/768714279","cluster_addr":"172.21.15.107:6821/768714279","heartbeat_back_addr":"172.21.15.107:6829/768714279","heartbeat_front_addr":"172.21.15.107:6825/768714279","state":["exists","up"]},{"osd":1,"uuid":"baef7a1f-8026-4fab-9e70-48465271034e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6801","nonce":2496355981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6803","nonce":2496355981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6807","nonce":2496355981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6805","nonce":2496355981}]},"public_addr":"172.21.15.107:6801/2496355981","cluster_addr":"172.21.15.107:6803/2496355981","heartbeat_back_addr":"172.21.15.107:6807/2496355981","heartbeat_front_addr":"172.21.15.107:6805/2496355981","state":["exists","up"]},{"osd":2,"uuid":"90e8cf24-7b54-484a-9d85-b44a9d3dfeab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6809","nonce":3097298224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6811","nonce":3097298224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6815","nonce":3097298224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6813","nonce":3097298224}]},"public_addr":"172.21.15.107:6809/3097298224","cluster_addr":"172.21.15.107:6811/3097298224","heartbeat_back_addr":"172.21.15.107:6815/3097298224","heartbeat_front_addr":"172.21.15.107:6813/3097298224","state":["exists","up"]},{"osd":3,"uuid":"54658731-2b9a-4d5e-ab13-7c1be7d34e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6820","nonce":3773076671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6824","nonce":3773076671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6831","nonce":3773076671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6828","nonce":3773076671}]},"public_addr":"172.21.15.107:6820/3773076671","cluster_addr":"172.21.15.107:6824/3773076671","heartbeat_back_addr":"172.21.15.107:6831/3773076671","heartbeat_front_addr":"172.21.15.107:6828/3773076671","state":["exists","up"]},{"osd":4,"uuid":"9a9dea13-8afe-485d-aa43-45eda691179b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6801","nonce":2747823542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6803","nonce":2747823542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6807","nonce":2747823542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6805","nonce":2747823542}]},"public_addr":"172.21.15.195:6801/2747823542","cluster_addr":"172.21.15.195:6803/2747823542","heartbeat_back_addr":"172.21.15.195:6807/2747823542","heartbeat_front_addr":"172.21.15.195:6805/2747823542","state":["exists","up"]},{"osd":5,"uuid":"db840d83-5a7d-468a-8aa2-1997945847c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6825","nonce":1484264540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6827","nonce":1484264540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6831","nonce":1484264540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6829","nonce":1484264540}]},"public_addr":"172.21.15.195:6825/1484264540","cluster_addr":"172.21.15.195:6827/1484264540","heartbeat_back_addr":"172.21.15.195:6831/1484264540","heartbeat_front_addr":"172.21.15.195:6829/1484264540","state":["exists","up"]},{"osd":6,"uuid":"38b5f165-bf91-467e-a615-4b4784337689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6811","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6813","nonce":2638471412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6817","nonce":2638471412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6823","nonce":2638471412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6821","nonce":2638471412}]},"public_addr":"172.21.15.195:6813/2638471412","cluster_addr":"172.21.15.195:6817/2638471412","heartbeat_back_addr":"172.21.15.195:6823/2638471412","heartbeat_front_addr":"172.21.15.195:6821/2638471412","state":["exists","up"]},{"osd":7,"uuid":"e5b3c596-494b-4df3-9437-bc2d52168ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6809","nonce":768189614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6812","nonce":768189614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6820","nonce":768189614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6816","nonce":768189614}]},"public_addr":"172.21.15.195:6809/768189614","cluster_addr":"172.21.15.195:6812/768189614","heartbeat_back_addr":"172.21.15.195:6820/768189614","heartbeat_front_addr":"172.21.15.195:6816/768189614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.209077+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.086601+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.221568+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.238524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.059477+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.148422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.079867+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.064916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T05:40:29.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.246+0000 7f3bcbc8f640 1 -- 172.21.15.107:0/1764966617 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3ba80668d0 msgr2=0x7f3ba8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:29.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.246+0000 7f3bcbc8f640 1 --2- 172.21.15.107:0/1764966617 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3ba80668d0 0x7f3ba8068d90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0001750 tx=0x7f3bc00011a0 comp rx=0 tx=0).stop 2023-12-04T05:40:29.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.246+0000 7f3bcbc8f640 1 -- 172.21.15.107:0/1764966617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bc40c9580 msgr2=0x7f3bc40ce570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:29.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.246+0000 7f3bcbc8f640 1 --2- 172.21.15.107:0/1764966617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bc40c9580 0x7f3bc40ce570 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40118d0 tx=0x7f3bb4011d90 comp rx=0 tx=0).stop 2023-12-04T05:40:29.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.246+0000 7f3bcbc8f640 1 -- 172.21.15.107:0/1764966617 shutdown_connections 2023-12-04T05:40:29.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.246+0000 7f3bcbc8f640 1 --2- 172.21.15.107:0/1764966617 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3ba80668d0 0x7f3ba8068d90 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:29.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.246+0000 7f3bcbc8f640 1 --2- 172.21.15.107:0/1764966617 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3bc40ceab0 0x7f3bc40c8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:29.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.246+0000 7f3bcbc8f640 1 --2- 172.21.15.107:0/1764966617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bc40c9580 0x7f3bc40ce570 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:29.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.246+0000 7f3bcbc8f640 1 --2- 172.21.15.107:0/1764966617 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3bb0003980 0x7f3bc415d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:29.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.246+0000 7f3bcbc8f640 1 -- 172.21.15.107:0/1764966617 >> 172.21.15.107:0/1764966617 conn(0x7f3bc406c6c0 msgr2=0x7f3bc4148f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:29.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.246+0000 7f3bcbc8f640 1 -- 172.21.15.107:0/1764966617 shutdown_connections 2023-12-04T05:40:29.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.246+0000 7f3bcbc8f640 1 -- 172.21.15.107:0/1764966617 wait complete. 2023-12-04T05:40:29.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:40:29.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4af068640 1 Processor -- start 2023-12-04T05:40:29.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4af068640 1 -- start start 2023-12-04T05:40:29.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4af068640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa4a80cb260 0x7fa4a80cb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:29.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4af068640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a815d530 con 0x7fa4a8154000 2023-12-04T05:40:29.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4af068640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a815d810 con 0x7fa4a80ca0f0 2023-12-04T05:40:29.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4af068640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a806bc20 con 0x7fa4a80cb260 2023-12-04T05:40:29.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4ace04640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa4a8154000 0x7fa4a806b510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42358/0 (socket says 172.21.15.107:42358) 2023-12-04T05:40:29.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4ace04640 1 -- 172.21.15.107:0/4107854833 learned_addr learned my addr 172.21.15.107:0/4107854833 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4ad605640 1 --2- 172.21.15.107:0/4107854833 >> v2:172.21.15.195:3300/0 conn(0x7fa4a80cb260 0x7fa4a80cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4a77fe640 1 -- 172.21.15.107:0/4107854833 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3006202029 0 0) 0x7fa4a815d530 con 0x7fa4a8154000 2023-12-04T05:40:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4a77fe640 1 -- 172.21.15.107:0/4107854833 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa498003530 con 0x7fa4a8154000 2023-12-04T05:40:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4a77fe640 1 -- 172.21.15.107:0/4107854833 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3905499796 0 0) 0x7fa498003530 con 0x7fa4a8154000 2023-12-04T05:40:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4a77fe640 1 -- 172.21.15.107:0/4107854833 >> v2:172.21.15.195:3300/0 conn(0x7fa4a80cb260 msgr2=0x7fa4a80cb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4a77fe640 1 --2- 172.21.15.107:0/4107854833 >> v2:172.21.15.195:3300/0 conn(0x7fa4a80cb260 0x7fa4a80cb670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4a77fe640 1 -- 172.21.15.107:0/4107854833 >> v1:172.21.15.195:6789/0 conn(0x7fa4a80ca0f0 legacy=0x7fa4a80cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4a77fe640 1 -- 172.21.15.107:0/4107854833 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a815daf0 con 0x7fa4a8154000 2023-12-04T05:40:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4a77fe640 1 -- 172.21.15.107:0/4107854833 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7fa49c001680 con 0x7fa4a8154000 2023-12-04T05:40:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4a77fe640 1 -- 172.21.15.107:0/4107854833 >> v1:172.21.15.107:6789/0 conn(0x7fa4a8154000 legacy=0x7fa4a806b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4a77fe640 1 --2- 172.21.15.107:0/4107854833 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa498003980 0x7fa498003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:29.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4a77fe640 1 --2- 172.21.15.107:0/4107854833 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4980047d0 0x7fa498004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:29.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4a77fe640 1 --2- 172.21.15.107:0/4107854833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa498005190 0x7fa49800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:29.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4a77fe640 1 -- 172.21.15.107:0/4107854833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49800c060 con 0x7fa498005190 2023-12-04T05:40:29.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4a77fe640 1 -- 172.21.15.107:0/4107854833 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49800c1a0 con 0x7fa4980047d0 2023-12-04T05:40:29.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4a77fe640 1 -- 172.21.15.107:0/4107854833 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa49800c360 con 0x7fa498003980 2023-12-04T05:40:29.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4ad605640 1 --2- 172.21.15.107:0/4107854833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa498005190 0x7fa49800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:29.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4a7fff640 1 --2- 172.21.15.107:0/4107854833 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4980047d0 0x7fa498004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:29.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4ace04640 1 --2- 172.21.15.107:0/4107854833 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa498003980 0x7fa498003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:29.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4ad605640 1 -- 172.21.15.107:0/4107854833 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa498003980 msgr2=0x7fa498003d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:29.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4ad605640 1 --2- 172.21.15.107:0/4107854833 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa498003980 0x7fa498003d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:29.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4ad605640 1 -- 172.21.15.107:0/4107854833 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4980047d0 msgr2=0x7fa498004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:29.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4ad605640 1 --2- 172.21.15.107:0/4107854833 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4980047d0 0x7fa498004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:29.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.382+0000 7fa4ad605640 1 -- 172.21.15.107:0/4107854833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa498003530 con 0x7fa498005190 2023-12-04T05:40:29.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.386+0000 7fa4ad605640 1 -- 172.21.15.107:0/4107854833 reap_dead start 2023-12-04T05:40:29.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.386+0000 7fa4ad605640 1 --2- 172.21.15.107:0/4107854833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa498005190 0x7fa49800b990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0007360 tx=0x7fa4a001ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0426a484f984263 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:29.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.386+0000 7fa4a77fe640 1 -- 172.21.15.107:0/4107854833 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a00a5d60 con 0x7fa498005190 2023-12-04T05:40:29.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.386+0000 7fa4a77fe640 1 -- 172.21.15.107:0/4107854833 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a00039a0 con 0x7fa498005190 2023-12-04T05:40:29.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.386+0000 7fa4a77fe640 1 -- 172.21.15.107:0/4107854833 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a0003d50 con 0x7fa498005190 2023-12-04T05:40:29.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.386+0000 7fa4af068640 1 -- 172.21.15.107:0/4107854833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa498005190 msgr2=0x7fa49800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:29.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.386+0000 7fa4af068640 1 --2- 172.21.15.107:0/4107854833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa498005190 0x7fa49800b990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0007360 tx=0x7fa4a001ce60 comp rx=0 tx=0).stop 2023-12-04T05:40:29.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.386+0000 7fa4af068640 1 -- 172.21.15.107:0/4107854833 shutdown_connections 2023-12-04T05:40:29.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.386+0000 7fa4af068640 1 --2- 172.21.15.107:0/4107854833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa498005190 0x7fa49800b990 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:29.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.386+0000 7fa4af068640 1 -- 172.21.15.107:0/4107854833 >> 172.21.15.107:0/4107854833 conn(0x7fa4a806c530 msgr2=0x7fa4a806c940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:29.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.386+0000 7fa4af068640 1 -- 172.21.15.107:0/4107854833 shutdown_connections 2023-12-04T05:40:29.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.386+0000 7fa4af068640 1 -- 172.21.15.107:0/4107854833 wait complete. 2023-12-04T05:40:29.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.386+0000 7fa4af068640 1 Processor -- start 2023-12-04T05:40:29.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.386+0000 7fa4af068640 1 -- start start 2023-12-04T05:40:29.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.386+0000 7fa4af068640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa498005190 0x7fa4a80ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:29.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.386+0000 7fa4af068640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4a815c250 0x7fa4a80b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:29.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.386+0000 7fa4af068640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa4a8151430 0x7fa4a80b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:29.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.386+0000 7fa4af068640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a80b4a30 con 0x7fa4a815c250 2023-12-04T05:40:29.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.386+0000 7fa4af068640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a8152bb0 con 0x7fa498005190 2023-12-04T05:40:29.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.386+0000 7fa4af068640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a815ca90 con 0x7fa4a8151430 2023-12-04T05:40:29.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.386+0000 7fa4a7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4a815c250 0x7fa4a80b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:29.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.390+0000 7fa4a7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4a815c250 0x7fa4a80b2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56382/0 (socket says 172.21.15.107:56382) 2023-12-04T05:40:29.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.390+0000 7fa4a7fff640 1 -- 172.21.15.107:0/1185680580 learned_addr learned my addr 172.21.15.107:0/1185680580 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:29.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.390+0000 7fa4ace04640 1 --2- 172.21.15.107:0/1185680580 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa498005190 0x7fa4a80ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:29.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.390+0000 7fa4ad605640 1 --2- 172.21.15.107:0/1185680580 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa4a8151430 0x7fa4a80b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:29.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.390+0000 7fa4a7fff640 1 -- 172.21.15.107:0/1185680580 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa4a8151430 msgr2=0x7fa4a80b32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:29.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.390+0000 7fa4a7fff640 1 --2- 172.21.15.107:0/1185680580 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa4a8151430 0x7fa4a80b32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:29.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.390+0000 7fa4a7fff640 1 -- 172.21.15.107:0/1185680580 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa498005190 msgr2=0x7fa4a80ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:29.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.390+0000 7fa4a7fff640 1 --2- 172.21.15.107:0/1185680580 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa498005190 0x7fa4a80ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:29.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.390+0000 7fa4a7fff640 1 -- 172.21.15.107:0/1185680580 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a815cd50 con 0x7fa4a815c250 2023-12-04T05:40:29.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.390+0000 7fa4ace04640 1 --2- 172.21.15.107:0/1185680580 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa498005190 0x7fa4a80ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:29.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.390+0000 7fa4a7fff640 1 --2- 172.21.15.107:0/1185680580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4a815c250 0x7fa4a80b2db0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa49400e5d0 tx=0x7fa49400ea90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:29.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.390+0000 7fa4a57fa640 1 -- 172.21.15.107:0/1185680580 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4940052e0 con 0x7fa4a815c250 2023-12-04T05:40:29.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.390+0000 7fa4a57fa640 1 -- 172.21.15.107:0/1185680580 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa494005460 con 0x7fa4a815c250 2023-12-04T05:40:29.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.390+0000 7fa4af068640 1 -- 172.21.15.107:0/1185680580 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a815d030 con 0x7fa4a815c250 2023-12-04T05:40:29.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.390+0000 7fa4a57fa640 1 -- 172.21.15.107:0/1185680580 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa494003840 con 0x7fa4a815c250 2023-12-04T05:40:29.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.390+0000 7fa4af068640 1 -- 172.21.15.107:0/1185680580 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a80b4370 con 0x7fa4a815c250 2023-12-04T05:40:29.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.390+0000 7fa4af068640 1 -- 172.21.15.107:0/1185680580 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a80b4b70 con 0x7fa4a815c250 2023-12-04T05:40:29.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.390+0000 7fa4a57fa640 1 -- 172.21.15.107:0/1185680580 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa4940039a0 con 0x7fa4a815c250 2023-12-04T05:40:29.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.390+0000 7fa4a57fa640 1 --2- 172.21.15.107:0/1185680580 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa48c066900 0x7fa48c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:29.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.390+0000 7fa4a57fa640 1 -- 172.21.15.107:0/1185680580 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4940841f0 con 0x7fa4a815c250 2023-12-04T05:40:29.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.390+0000 7fa4ace04640 1 --2- 172.21.15.107:0/1185680580 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa48c066900 0x7fa48c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:29.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.390+0000 7fa4ace04640 1 --2- 172.21.15.107:0/1185680580 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa48c066900 0x7fa48c068dc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa49c0017b0 tx=0x7fa49c000f40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:29.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.394+0000 7fa4a57fa640 1 -- 172.21.15.107:0/1185680580 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa4a80b4b70 con 0x7fa4a815c250 2023-12-04T05:40:29.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.570+0000 7fa4af068640 1 -- 172.21.15.107:0/1185680580 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4a80b4b70 con 0x7fa4a815c250 2023-12-04T05:40:29.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.570+0000 7fa4a57fa640 1 -- 172.21.15.107:0/1185680580 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+6564 (secure 0 0 0) 0x7fa4a80b4b70 con 0x7fa4a815c250 2023-12-04T05:40:29.573 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:29.573 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:40:23.915736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":16,"state":"up:creating","state_seq":1,"addr":"172.21.15.195:6836/3769196693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":3769196693},{"type":"v1","addr":"172.21.15.195:6836","nonce":3769196693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:40:29.573 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 16 2023-12-04T05:40:29.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.570+0000 7fa4af068640 1 -- 172.21.15.107:0/1185680580 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa48c066900 msgr2=0x7fa48c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:29.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.570+0000 7fa4af068640 1 --2- 172.21.15.107:0/1185680580 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa48c066900 0x7fa48c068dc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa49c0017b0 tx=0x7fa49c000f40 comp rx=0 tx=0).stop 2023-12-04T05:40:29.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.574+0000 7fa4af068640 1 -- 172.21.15.107:0/1185680580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4a815c250 msgr2=0x7fa4a80b2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:29.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.574+0000 7fa4af068640 1 --2- 172.21.15.107:0/1185680580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4a815c250 0x7fa4a80b2db0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa49400e5d0 tx=0x7fa49400ea90 comp rx=0 tx=0).stop 2023-12-04T05:40:29.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.574+0000 7fa4af068640 1 -- 172.21.15.107:0/1185680580 shutdown_connections 2023-12-04T05:40:29.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.574+0000 7fa4af068640 1 --2- 172.21.15.107:0/1185680580 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa48c066900 0x7fa48c068dc0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:29.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.574+0000 7fa4af068640 1 --2- 172.21.15.107:0/1185680580 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa4a8151430 0x7fa4a80b32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:29.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.574+0000 7fa4af068640 1 --2- 172.21.15.107:0/1185680580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4a815c250 0x7fa4a80b2db0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:29.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.574+0000 7fa4af068640 1 --2- 172.21.15.107:0/1185680580 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa498005190 0x7fa4a80ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:29.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.574+0000 7fa4af068640 1 -- 172.21.15.107:0/1185680580 >> 172.21.15.107:0/1185680580 conn(0x7fa4a806c530 msgr2=0x7fa4a806b590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:29.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.574+0000 7fa4af068640 1 -- 172.21.15.107:0/1185680580 shutdown_connections 2023-12-04T05:40:29.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:29.574+0000 7fa4af068640 1 -- 172.21.15.107:0/1185680580 wait complete. 2023-12-04T05:40:29.899 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:40:29.902 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:40:30.196 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-04T05:39:59.032019+0000', 'modified': '2023-12-04T05:40:23.915736+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4288, 'mds_1': 4279, 'mds_2': 4263, 'mds_3': 4260, 'mds_4': 4250}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4250': {'gid': 4250, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.195:6836/3769196693', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6834', 'nonce': 3769196693}, {'type': 'v1', 'addr': '172.21.15.195:6836', 'nonce': 3769196693}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4260': {'gid': 4260, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.107:6836/3444851515', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6834', 'nonce': 3444851515}, {'type': 'v1', 'addr': '172.21.15.107:6836', 'nonce': 3444851515}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4263': {'gid': 4263, 'name': 'c', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.107:6837/2210527429', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6835', 'nonce': 2210527429}, {'type': 'v1', 'addr': '172.21.15.107:6837', 'nonce': 2210527429}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4279': {'gid': 4279, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.195:6833/569093002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6832', 'nonce': 569093002}, {'type': 'v1', 'addr': '172.21.15.195:6833', 'nonce': 569093002}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4288': {'gid': 4288, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.195:6837/3412865477', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6835', 'nonce': 3412865477}, {'type': 'v1', 'addr': '172.21.15.195:6837', 'nonce': 3412865477}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-12-04T05:40:30.196 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4250:up:creating 2023-12-04T05:40:30.261 INFO:tasks.ceph.mds.f.smithi195.stderr:2023-12-04T05:40:30.255+0000 7f5bdadb8640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-12-04T05:40:31.197 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:40:31.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f39907b0640 1 Processor -- start 2023-12-04T05:40:31.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f39907b0640 1 -- start start 2023-12-04T05:40:31.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f39907b0640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f39880ca2d0 0x7f3988072570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:31.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f39907b0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39880c9570 con 0x7f39880ca6b0 2023-12-04T05:40:31.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f39907b0640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39880c8c80 con 0x7f3988072ab0 2023-12-04T05:40:31.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f39907b0640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3988073290 con 0x7f39880ca2d0 2023-12-04T05:40:31.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398dd4b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f39880ca6b0 0x7f39880cd510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42370/0 (socket says 172.21.15.107:42370) 2023-12-04T05:40:31.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398dd4b640 1 -- 172.21.15.107:0/2388822231 learned_addr learned my addr 172.21.15.107:0/2388822231 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:31.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398e54c640 1 --2- 172.21.15.107:0/2388822231 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f39880ca2d0 0x7f3988072570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:31.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398d54a640 1 -- 172.21.15.107:0/2388822231 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343773099 0 0) 0x7f39880c9570 con 0x7f39880ca6b0 2023-12-04T05:40:31.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398d54a640 1 -- 172.21.15.107:0/2388822231 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3974003530 con 0x7f39880ca6b0 2023-12-04T05:40:31.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398d54a640 1 -- 172.21.15.107:0/2388822231 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 961472607 0 0) 0x7f3974003530 con 0x7f39880ca6b0 2023-12-04T05:40:31.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398d54a640 1 -- 172.21.15.107:0/2388822231 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f39880ca2d0 msgr2=0x7f3988072570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:31.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398d54a640 1 --2- 172.21.15.107:0/2388822231 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f39880ca2d0 0x7f3988072570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:31.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398d54a640 1 -- 172.21.15.107:0/2388822231 >> v1:172.21.15.195:6789/0 conn(0x7f3988072ab0 legacy=0x7f3988151960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:31.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398d54a640 1 -- 172.21.15.107:0/2388822231 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3988072fb0 con 0x7f39880ca6b0 2023-12-04T05:40:31.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398d54a640 1 -- 172.21.15.107:0/2388822231 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f3978001680 con 0x7f39880ca6b0 2023-12-04T05:40:31.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398d54a640 1 -- 172.21.15.107:0/2388822231 >> v1:172.21.15.107:6789/0 conn(0x7f39880ca6b0 legacy=0x7f39880cd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398d54a640 1 --2- 172.21.15.107:0/2388822231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39740039b0 0x7f3974003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398d54a640 1 --2- 172.21.15.107:0/2388822231 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f39880ca2d0 0x7f39740047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398d54a640 1 --2- 172.21.15.107:0/2388822231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3974009980 0x7f3974004d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398d54a640 1 -- 172.21.15.107:0/2388822231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3988073290 con 0x7f39740039b0 2023-12-04T05:40:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398d54a640 1 -- 172.21.15.107:0/2388822231 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3974005510 con 0x7f3974009980 2023-12-04T05:40:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398d54a640 1 -- 172.21.15.107:0/2388822231 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3974014590 con 0x7f39880ca2d0 2023-12-04T05:40:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398e54c640 1 --2- 172.21.15.107:0/2388822231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39740039b0 0x7f3974003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398ed4d640 1 --2- 172.21.15.107:0/2388822231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3974009980 0x7f3974004d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398e54c640 1 -- 172.21.15.107:0/2388822231 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f39880ca2d0 msgr2=0x7f39740047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398dd4b640 1 --2- 172.21.15.107:0/2388822231 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f39880ca2d0 0x7f39740047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398e54c640 1 --2- 172.21.15.107:0/2388822231 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f39880ca2d0 0x7f39740047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398e54c640 1 -- 172.21.15.107:0/2388822231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3974009980 msgr2=0x7f3974004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398e54c640 1 --2- 172.21.15.107:0/2388822231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3974009980 0x7f3974004d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398e54c640 1 -- 172.21.15.107:0/2388822231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3974003530 con 0x7f39740039b0 2023-12-04T05:40:31.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.326+0000 7f398dd4b640 1 --2- 172.21.15.107:0/2388822231 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f39880ca2d0 0x7f39740047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:31.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.330+0000 7f398e54c640 1 --2- 172.21.15.107:0/2388822231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39740039b0 0x7f3974003dd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3984009ae0 tx=0x7f398401c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b34239bc34b2eb0e server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:31.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.330+0000 7f398d54a640 1 -- 172.21.15.107:0/2388822231 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39840a6470 con 0x7f39740039b0 2023-12-04T05:40:31.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.330+0000 7f398d54a640 1 -- 172.21.15.107:0/2388822231 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39840a65d0 con 0x7f39740039b0 2023-12-04T05:40:31.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.330+0000 7f398d54a640 1 -- 172.21.15.107:0/2388822231 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39840a6980 con 0x7f39740039b0 2023-12-04T05:40:31.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.330+0000 7f39907b0640 1 -- 172.21.15.107:0/2388822231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39740039b0 msgr2=0x7f3974003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:31.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.330+0000 7f39907b0640 1 --2- 172.21.15.107:0/2388822231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39740039b0 0x7f3974003dd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3984009ae0 tx=0x7f398401c870 comp rx=0 tx=0).stop 2023-12-04T05:40:31.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.330+0000 7f398ed4d640 1 -- 172.21.15.107:0/2388822231 reap_dead start 2023-12-04T05:40:31.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.330+0000 7f39907b0640 1 -- 172.21.15.107:0/2388822231 shutdown_connections 2023-12-04T05:40:31.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.330+0000 7f39907b0640 1 -- 172.21.15.107:0/2388822231 >> 172.21.15.107:0/2388822231 conn(0x7f39880663a0 msgr2=0x7f39880667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:31.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.330+0000 7f39907b0640 1 -- 172.21.15.107:0/2388822231 shutdown_connections 2023-12-04T05:40:31.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.330+0000 7f39907b0640 1 -- 172.21.15.107:0/2388822231 wait complete. 2023-12-04T05:40:31.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.330+0000 7f39907b0640 1 Processor -- start 2023-12-04T05:40:31.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f39907b0640 1 -- start start 2023-12-04T05:40:31.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f39907b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39740039b0 0x7f398814fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:31.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f39907b0640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3988141cb0 0x7f398806faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:31.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f39907b0640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39881eed10 0x7f39881ef190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:31.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f39907b0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39881ee880 con 0x7f39740039b0 2023-12-04T05:40:31.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f39907b0640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39880ccb40 con 0x7f39881eed10 2023-12-04T05:40:31.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f39907b0640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39881ef6d0 con 0x7f3988141cb0 2023-12-04T05:40:31.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f398e54c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39740039b0 0x7f398814fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:31.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f398e54c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39740039b0 0x7f398814fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56402/0 (socket says 172.21.15.107:56402) 2023-12-04T05:40:31.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f398e54c640 1 -- 172.21.15.107:0/2169393237 learned_addr learned my addr 172.21.15.107:0/2169393237 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:31.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f398dd4b640 1 --2- 172.21.15.107:0/2169393237 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3988141cb0 0x7f398806faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:31.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f398ed4d640 1 --2- 172.21.15.107:0/2169393237 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39881eed10 0x7f39881ef190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:31.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f398e54c640 1 -- 172.21.15.107:0/2169393237 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3988141cb0 msgr2=0x7f398806faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:31.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f398e54c640 1 --2- 172.21.15.107:0/2169393237 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3988141cb0 0x7f398806faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:31.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f398e54c640 1 -- 172.21.15.107:0/2169393237 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39881eed10 msgr2=0x7f39881ef190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:31.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f398e54c640 1 --2- 172.21.15.107:0/2169393237 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39881eed10 0x7f39881ef190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:31.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f398e54c640 1 -- 172.21.15.107:0/2169393237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398814dcc0 con 0x7f39740039b0 2023-12-04T05:40:31.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f398ed4d640 1 --2- 172.21.15.107:0/2169393237 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39881eed10 0x7f39881ef190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:31.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f398e54c640 1 --2- 172.21.15.107:0/2169393237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39740039b0 0x7f398814fd80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3984009ae0 tx=0x7f3974005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:31.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f397effd640 1 -- 172.21.15.107:0/2169393237 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39840a6dc0 con 0x7f39740039b0 2023-12-04T05:40:31.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f397effd640 1 -- 172.21.15.107:0/2169393237 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39840a7450 con 0x7f39740039b0 2023-12-04T05:40:31.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f39907b0640 1 -- 172.21.15.107:0/2169393237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f398814df80 con 0x7f39740039b0 2023-12-04T05:40:31.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f397effd640 1 -- 172.21.15.107:0/2169393237 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39840a77e0 con 0x7f39740039b0 2023-12-04T05:40:31.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f39907b0640 1 -- 172.21.15.107:0/2169393237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39880cd090 con 0x7f39740039b0 2023-12-04T05:40:31.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f39907b0640 1 -- 172.21.15.107:0/2169393237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39880c88b0 con 0x7f39740039b0 2023-12-04T05:40:31.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f397effd640 1 -- 172.21.15.107:0/2169393237 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f39840a7940 con 0x7f39740039b0 2023-12-04T05:40:31.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f397effd640 1 --2- 172.21.15.107:0/2169393237 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f396c066900 0x7f396c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:31.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f397effd640 1 -- 172.21.15.107:0/2169393237 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f398411e000 con 0x7f39740039b0 2023-12-04T05:40:31.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.334+0000 7f398dd4b640 1 --2- 172.21.15.107:0/2169393237 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f396c066900 0x7f396c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:31.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.338+0000 7f398dd4b640 1 --2- 172.21.15.107:0/2169393237 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f396c066900 0x7f396c068dc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3988142400 tx=0x7f3978001500 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:31.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.342+0000 7f397effd640 1 -- 172.21.15.107:0/2169393237 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f39840eb240 con 0x7f39740039b0 2023-12-04T05:40:31.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.522+0000 7f39907b0640 1 -- 172.21.15.107:0/2169393237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3988067ec0 con 0x7f39740039b0 2023-12-04T05:40:31.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.522+0000 7f397effd640 1 -- 172.21.15.107:0/2169393237 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6563 (secure 0 0 0) 0x7f39840ee860 con 0x7f39740039b0 2023-12-04T05:40:31.527 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:31.527 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:40:28.948661+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":10,"addr":"172.21.15.195:6836/3769196693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":3769196693},{"type":"v1","addr":"172.21.15.195:6836","nonce":3769196693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:40:31.527 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-12-04T05:40:31.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.526+0000 7f39907b0640 1 -- 172.21.15.107:0/2169393237 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f396c066900 msgr2=0x7f396c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:31.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.526+0000 7f39907b0640 1 --2- 172.21.15.107:0/2169393237 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f396c066900 0x7f396c068dc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3988142400 tx=0x7f3978001500 comp rx=0 tx=0).stop 2023-12-04T05:40:31.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.526+0000 7f39907b0640 1 -- 172.21.15.107:0/2169393237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39740039b0 msgr2=0x7f398814fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:31.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.526+0000 7f39907b0640 1 --2- 172.21.15.107:0/2169393237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39740039b0 0x7f398814fd80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3984009ae0 tx=0x7f3974005800 comp rx=0 tx=0).stop 2023-12-04T05:40:31.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.526+0000 7f39907b0640 1 -- 172.21.15.107:0/2169393237 shutdown_connections 2023-12-04T05:40:31.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.526+0000 7f39907b0640 1 --2- 172.21.15.107:0/2169393237 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f396c066900 0x7f396c068dc0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:31.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.526+0000 7f39907b0640 1 --2- 172.21.15.107:0/2169393237 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f39881eed10 0x7f39881ef190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:31.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.526+0000 7f39907b0640 1 --2- 172.21.15.107:0/2169393237 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3988141cb0 0x7f398806faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:31.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.526+0000 7f39907b0640 1 --2- 172.21.15.107:0/2169393237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39740039b0 0x7f398814fd80 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:31.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.526+0000 7f39907b0640 1 -- 172.21.15.107:0/2169393237 >> 172.21.15.107:0/2169393237 conn(0x7f39880663a0 msgr2=0x7f39880cd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:31.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.526+0000 7f39907b0640 1 -- 172.21.15.107:0/2169393237 shutdown_connections 2023-12-04T05:40:31.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:31.526+0000 7f39907b0640 1 -- 172.21.15.107:0/2169393237 wait complete. 2023-12-04T05:40:32.148 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-04T05:39:59.032019+0000', 'modified': '2023-12-04T05:40:28.948661+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4288, 'mds_1': 4279, 'mds_2': 4263, 'mds_3': 4260, 'mds_4': 4250}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4250': {'gid': 4250, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.195:6836/3769196693', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6834', 'nonce': 3769196693}, {'type': 'v1', 'addr': '172.21.15.195:6836', 'nonce': 3769196693}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4260': {'gid': 4260, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.107:6836/3444851515', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6834', 'nonce': 3444851515}, {'type': 'v1', 'addr': '172.21.15.107:6836', 'nonce': 3444851515}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4263': {'gid': 4263, 'name': 'c', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.107:6837/2210527429', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6835', 'nonce': 2210527429}, {'type': 'v1', 'addr': '172.21.15.107:6837', 'nonce': 2210527429}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4279': {'gid': 4279, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.195:6833/569093002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6832', 'nonce': 569093002}, {'type': 'v1', 'addr': '172.21.15.195:6833', 'nonce': 569093002}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4288': {'gid': 4288, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.195:6837/3412865477', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6835', 'nonce': 3412865477}, {'type': 'v1', 'addr': '172.21.15.195:6837', 'nonce': 3412865477}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-12-04T05:40:32.148 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2023-12-04T05:40:32.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-12-04T05:40:32.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.274+0000 7fb73adbb640 1 Processor -- start 2023-12-04T05:40:32.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.274+0000 7fb73adbb640 1 -- start start 2023-12-04T05:40:32.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.274+0000 7fb73adbb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb734072110 0x7fb734158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:32.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.274+0000 7fb73adbb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb7340724f0 0x7fb7340c7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:32.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.274+0000 7fb73adbb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb734158da0 con 0x7fb7340c56f0 2023-12-04T05:40:32.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.274+0000 7fb73adbb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7341583d0 con 0x7fb7340724f0 2023-12-04T05:40:32.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.274+0000 7fb73adbb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7340c6170 con 0x7fb734072110 2023-12-04T05:40:32.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.274+0000 7fb733fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb7340724f0 0x7fb7340c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:32.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.274+0000 7fb733fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb7340724f0 0x7fb7340c7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56416/0 (socket says 172.21.15.107:56416) 2023-12-04T05:40:32.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.274+0000 7fb733fff640 1 -- 172.21.15.107:0/619087265 learned_addr learned my addr 172.21.15.107:0/619087265 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:32.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.278+0000 7fb738b57640 1 --2- 172.21.15.107:0/619087265 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb734072110 0x7fb734158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:32.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.278+0000 7fb733fff640 1 -- 172.21.15.107:0/619087265 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb734072110 msgr2=0x7fb734158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:32.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.278+0000 7fb733fff640 1 --2- 172.21.15.107:0/619087265 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb734072110 0x7fb734158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:32.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.278+0000 7fb733fff640 1 -- 172.21.15.107:0/619087265 >> v1:172.21.15.195:6789/0 conn(0x7fb7340c56f0 legacy=0x7fb734140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:32.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.278+0000 7fb733fff640 1 -- 172.21.15.107:0/619087265 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb734157da0 con 0x7fb7340724f0 2023-12-04T05:40:32.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.278+0000 7fb738b57640 1 --2- 172.21.15.107:0/619087265 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb734072110 0x7fb734158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:32.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.278+0000 7fb733fff640 1 --2- 172.21.15.107:0/619087265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340724f0 0x7fb7340c7420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb728009870 tx=0x7fb72801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a83d89fc1f6eb83 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:32.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.278+0000 7fb7337fe640 1 -- 172.21.15.107:0/619087265 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7280ac410 con 0x7fb7340724f0 2023-12-04T05:40:32.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.278+0000 7fb7337fe640 1 -- 172.21.15.107:0/619087265 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7280ac570 con 0x7fb7340724f0 2023-12-04T05:40:32.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.278+0000 7fb7337fe640 1 -- 172.21.15.107:0/619087265 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7280ac900 con 0x7fb7340724f0 2023-12-04T05:40:32.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.278+0000 7fb73adbb640 1 -- 172.21.15.107:0/619087265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340724f0 msgr2=0x7fb7340c7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:32.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.278+0000 7fb73adbb640 1 --2- 172.21.15.107:0/619087265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340724f0 0x7fb7340c7420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb728009870 tx=0x7fb72801c500 comp rx=0 tx=0).stop 2023-12-04T05:40:32.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.278+0000 7fb73adbb640 1 -- 172.21.15.107:0/619087265 shutdown_connections 2023-12-04T05:40:32.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.278+0000 7fb73adbb640 1 --2- 172.21.15.107:0/619087265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340724f0 0x7fb7340c7420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:32.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.278+0000 7fb73adbb640 1 --2- 172.21.15.107:0/619087265 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb734072110 0x7fb734158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:32.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.278+0000 7fb73adbb640 1 -- 172.21.15.107:0/619087265 >> 172.21.15.107:0/619087265 conn(0x7fb7340c7ff0 msgr2=0x7fb7340c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:32.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.278+0000 7fb73adbb640 1 -- 172.21.15.107:0/619087265 shutdown_connections 2023-12-04T05:40:32.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.278+0000 7fb73adbb640 1 -- 172.21.15.107:0/619087265 wait complete. 2023-12-04T05:40:32.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb73adbb640 1 Processor -- start 2023-12-04T05:40:32.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb73adbb640 1 -- start start 2023-12-04T05:40:32.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb73adbb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb734072110 0x7fb7341f4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:32.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb73adbb640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7340724f0 0x7fb73412ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:32.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb73adbb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb7340c56f0 0x7fb7341367a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:32.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb73adbb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7341f4fb0 con 0x7fb734072110 2023-12-04T05:40:32.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb73adbb640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb734136310 con 0x7fb7340724f0 2023-12-04T05:40:32.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb73adbb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7341f1a40 con 0x7fb7340c56f0 2023-12-04T05:40:32.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb738b57640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb734072110 0x7fb7341f4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:32.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb738b57640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb734072110 0x7fb7341f4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56420/0 (socket says 172.21.15.107:56420) 2023-12-04T05:40:32.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb738b57640 1 -- 172.21.15.107:0/3873061897 learned_addr learned my addr 172.21.15.107:0/3873061897 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:32.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb739358640 1 --2- 172.21.15.107:0/3873061897 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb7340c56f0 0x7fb7341367a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:32.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb733fff640 1 --2- 172.21.15.107:0/3873061897 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7340724f0 0x7fb73412ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:32.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb738b57640 1 -- 172.21.15.107:0/3873061897 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb7340c56f0 msgr2=0x7fb7341367a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:32.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb738b57640 1 --2- 172.21.15.107:0/3873061897 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb7340c56f0 0x7fb7341367a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:32.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb738b57640 1 -- 172.21.15.107:0/3873061897 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7340724f0 msgr2=0x7fb73412ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:32.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb738b57640 1 --2- 172.21.15.107:0/3873061897 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7340724f0 0x7fb73412ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:32.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb738b57640 1 -- 172.21.15.107:0/3873061897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7280094d0 con 0x7fb734072110 2023-12-04T05:40:32.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb738b57640 1 --2- 172.21.15.107:0/3873061897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb734072110 0x7fb7341f4a70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb72400c7f0 tx=0x7fb72400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:32.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb7317fa640 1 -- 172.21.15.107:0/3873061897 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724007820 con 0x7fb734072110 2023-12-04T05:40:32.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb7317fa640 1 -- 172.21.15.107:0/3873061897 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb724007980 con 0x7fb734072110 2023-12-04T05:40:32.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb73adbb640 1 -- 172.21.15.107:0/3873061897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb734156ee0 con 0x7fb734072110 2023-12-04T05:40:32.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb7317fa640 1 -- 172.21.15.107:0/3873061897 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724007d30 con 0x7fb734072110 2023-12-04T05:40:32.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.282+0000 7fb73adbb640 1 -- 172.21.15.107:0/3873061897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7341f11a0 con 0x7fb734072110 2023-12-04T05:40:32.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb7317fa640 1 -- 172.21.15.107:0/3873061897 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fb7240113f0 con 0x7fb734072110 2023-12-04T05:40:32.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb7317fa640 1 --2- 172.21.15.107:0/3873061897 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb720066900 0x7fb720068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:32.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb733fff640 1 --2- 172.21.15.107:0/3873061897 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb720066900 0x7fb720068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:32.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb7317fa640 1 -- 172.21.15.107:0/3873061897 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb724084720 con 0x7fb734072110 2023-12-04T05:40:32.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb733fff640 1 --2- 172.21.15.107:0/3873061897 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb720066900 0x7fb720068dc0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb7280ae000 tx=0x7fb728008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:32.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb73deed000 1 Processor -- start 2023-12-04T05:40:32.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb73deed000 1 -- start start 2023-12-04T05:40:32.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb73deed000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda040 0x559db4dda420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:32.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb73deed000 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559db4dda960 0x559db4e14a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:32.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb73deed000 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559db4e14fc0 0x559db4e153a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:32.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb73deed000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x559db4c9b110 con 0x559db4dda040 2023-12-04T05:40:32.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb73deed000 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x559db4c2b9a0 con 0x559db4dda960 2023-12-04T05:40:32.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb73deed000 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x559db4c28ed0 con 0x559db4e14fc0 2023-12-04T05:40:32.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb738b57640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda040 0x559db4dda420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:32.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb738b57640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda040 0x559db4dda420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56434/0 (socket says 172.21.15.107:56434) 2023-12-04T05:40:32.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb738b57640 1 -- 172.21.15.107:0/326510726 learned_addr learned my addr 172.21.15.107:0/326510726 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:32.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb739358640 1 --2- 172.21.15.107:0/326510726 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559db4e14fc0 0x559db4e153a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:32.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb733fff640 1 --2- 172.21.15.107:0/326510726 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559db4dda960 0x559db4e14a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:32.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb738b57640 1 -- 172.21.15.107:0/326510726 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559db4e14fc0 msgr2=0x559db4e153a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:32.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb738b57640 1 --2- 172.21.15.107:0/326510726 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559db4e14fc0 0x559db4e153a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:32.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb738b57640 1 -- 172.21.15.107:0/326510726 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559db4dda960 msgr2=0x559db4e14a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:32.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb738b57640 1 --2- 172.21.15.107:0/326510726 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559db4dda960 0x559db4e14a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:32.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb738b57640 1 -- 172.21.15.107:0/326510726 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559db491c890 con 0x559db4dda040 2023-12-04T05:40:32.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb739358640 1 --2- 172.21.15.107:0/326510726 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559db4e14fc0 0x559db4e153a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:32.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb733fff640 1 --2- 172.21.15.107:0/326510726 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559db4dda960 0x559db4e14a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:32.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb738b57640 1 --2- 172.21.15.107:0/326510726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda040 0x559db4dda420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb72405d3e0 tx=0x7fb724055070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7285d76598214f1 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:32.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb71a7fc640 1 -- 172.21.15.107:0/326510726 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72405d910 con 0x559db4dda040 2023-12-04T05:40:32.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb71a7fc640 1 -- 172.21.15.107:0/326510726 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7240539b0 con 0x559db4dda040 2023-12-04T05:40:32.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb71a7fc640 1 -- 172.21.15.107:0/326510726 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72405d480 con 0x559db4dda040 2023-12-04T05:40:32.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb73deed000 1 -- 172.21.15.107:0/326510726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda040 msgr2=0x559db4dda420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:32.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb73deed000 1 --2- 172.21.15.107:0/326510726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda040 0x559db4dda420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb72405d3e0 tx=0x7fb724055070 comp rx=0 tx=0).stop 2023-12-04T05:40:32.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb73deed000 1 -- 172.21.15.107:0/326510726 shutdown_connections 2023-12-04T05:40:32.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb73deed000 1 --2- 172.21.15.107:0/326510726 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559db4e14fc0 0x559db4e153a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:32.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb73deed000 1 --2- 172.21.15.107:0/326510726 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559db4dda960 0x559db4e14a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:32.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb73deed000 1 --2- 172.21.15.107:0/326510726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda040 0x559db4dda420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:32.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.286+0000 7fb73deed000 1 -- 172.21.15.107:0/326510726 >> 172.21.15.107:0/326510726 conn(0x559db4e11840 msgr2=0x559db4e13c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:32.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb73deed000 1 -- 172.21.15.107:0/326510726 shutdown_connections 2023-12-04T05:40:32.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb73deed000 1 -- 172.21.15.107:0/326510726 wait complete. 2023-12-04T05:40:32.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb73deed000 1 -- start start 2023-12-04T05:40:32.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb73deed000 1 Processor -- start 2023-12-04T05:40:32.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb73deed000 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559db4dda040 0x559db4e0fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:32.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb73deed000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda960 0x559db4e102f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:32.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb73deed000 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559db4e14fc0 0x559db4dd6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:32.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb73deed000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x559db4c247c0 con 0x559db4dda960 2023-12-04T05:40:32.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb73deed000 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x559db4bdcbf0 con 0x559db4dda040 2023-12-04T05:40:32.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb73deed000 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x559db4be2a90 con 0x559db4e14fc0 2023-12-04T05:40:32.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb73deed000 10 client.0 _ll_register_callbacks cb 0x559db4dbfe20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-04T05:40:32.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb733fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda960 0x559db4e102f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:32.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb733fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda960 0x559db4e102f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56438/0 (socket says 172.21.15.107:56438) 2023-12-04T05:40:32.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb733fff640 1 -- 172.21.15.107:0/3738532173 learned_addr learned my addr 172.21.15.107:0/3738532173 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:32.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb738b57640 1 --2- 172.21.15.107:0/3738532173 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559db4dda040 0x559db4e0fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:32.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb739358640 1 --2- 172.21.15.107:0/3738532173 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559db4e14fc0 0x559db4dd6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:32.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb733fff640 1 -- 172.21.15.107:0/3738532173 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559db4e14fc0 msgr2=0x559db4dd6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:32.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb733fff640 1 --2- 172.21.15.107:0/3738532173 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559db4e14fc0 0x559db4dd6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:32.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb733fff640 1 -- 172.21.15.107:0/3738532173 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559db4dda040 msgr2=0x559db4e0fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:32.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb733fff640 1 --2- 172.21.15.107:0/3738532173 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559db4dda040 0x559db4e0fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:32.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb733fff640 1 -- 172.21.15.107:0/3738532173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7341f11a0 con 0x559db4dda960 2023-12-04T05:40:32.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb733fff640 1 --2- 172.21.15.107:0/3738532173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda960 0x559db4e102f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb72801c540 tx=0x7fb72801ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=288c089ee0e487da server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:32.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb7137fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-12-04T05:40:32.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb7137fe640 1 -- 172.21.15.107:0/3738532173 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7280bc070 con 0x559db4dda960 2023-12-04T05:40:32.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb7137fe640 1 -- 172.21.15.107:0/3738532173 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb7280b78f0 con 0x559db4dda960 2023-12-04T05:40:32.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb73deed000 1 -- 172.21.15.107:0/3738532173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559db4be6470 con 0x559db4dda960 2023-12-04T05:40:32.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb7137fe640 1 -- 172.21.15.107:0/3738532173 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7280b7ad0 con 0x559db4dda960 2023-12-04T05:40:32.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb73deed000 10 client.4656 fetch_fsmap learned FSMap version 17 2023-12-04T05:40:32.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb73deed000 1 -- 172.21.15.107:0/3738532173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x559db4be5190 con 0x559db4dda960 2023-12-04T05:40:32.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb7137fe640 1 -- 172.21.15.107:0/3738532173 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559db4be6470 con 0x559db4dda960 2023-12-04T05:40:32.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb7137fe640 1 -- 172.21.15.107:0/3738532173 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 17) v1 ==== 4492+0+0 (secure 0 0 0) 0x7fb7280b7c30 con 0x559db4dda960 2023-12-04T05:40:32.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb73deed000 10 client.4656 fetch_fsmap finished waiting for FSMap version 17 2023-12-04T05:40:32.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb73deed000 10 client.4656 resolve_mds: resolved name 'b' to daemon mds.b 2023-12-04T05:40:32.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb73deed000 20 client.4656 populate_metadata read hostname 'smithi107' 2023-12-04T05:40:32.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb73deed000 1 --2- 172.21.15.107:0/3738532173 >> [v2:172.21.15.195:6834/3769196693,v1:172.21.15.195:6836/3769196693] conn(0x559db4dd7110 0x559db4dd74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:32.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb73deed000 4 client.4656 mds_command: new command op to 4250 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-12-04T05:40:32.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb73deed000 1 -- 172.21.15.107:0/3738532173 --> [v2:172.21.15.195:6834/3769196693,v1:172.21.15.195:6836/3769196693] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559db4be9b50 con 0x559db4dd7110 2023-12-04T05:40:32.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb738b57640 1 --2- 172.21.15.107:0/3738532173 >> [v2:172.21.15.195:6834/3769196693,v1:172.21.15.195:6836/3769196693] conn(0x559db4dd7110 0x559db4dd74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:32.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb738b57640 1 --2- 172.21.15.107:0/3738532173 >> [v2:172.21.15.195:6834/3769196693,v1:172.21.15.195:6836/3769196693] conn(0x559db4dd7110 0x559db4dd74f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c5e1bc07b67b8c22 server_cookie=2552e3bf34813bcb in_seq=0 out_seq=0 2023-12-04T05:40:32.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:32.290+0000 7fb7137fe640 10 client.4656 ms_handle_connect on v2:172.21.15.195:6834/3769196693 2023-12-04T05:40:34.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.954+0000 7fb7137fe640 1 -- 172.21.15.107:0/3738532173 <== mds.4 v2:172.21.15.195:6834/3769196693 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559db4be9b50 con 0x559db4dd7110 2023-12-04T05:40:34.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.954+0000 7fb7137fe640 10 client.4656 handle_command_reply: tid=0 2023-12-04T05:40:34.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.954+0000 7fb73deed000 1 client.4656 shutdown 2023-12-04T05:40:34.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.954+0000 7fb73deed000 1 -- 172.21.15.107:0/3738532173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda960 msgr2=0x559db4e102f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:34.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.954+0000 7fb73deed000 1 --2- 172.21.15.107:0/3738532173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda960 0x559db4e102f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb72801c540 tx=0x7fb72801ad40 comp rx=0 tx=0).stop 2023-12-04T05:40:34.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.954+0000 7fb73deed000 1 -- 172.21.15.107:0/3738532173 shutdown_connections 2023-12-04T05:40:34.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.954+0000 7fb73deed000 1 --2- 172.21.15.107:0/3738532173 >> [v2:172.21.15.195:6834/3769196693,v1:172.21.15.195:6836/3769196693] conn(0x559db4dd7110 0x559db4dd74f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:34.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.954+0000 7fb73deed000 1 --2- 172.21.15.107:0/3738532173 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559db4e14fc0 0x559db4dd6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:34.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.954+0000 7fb73deed000 1 --2- 172.21.15.107:0/3738532173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda960 0x559db4e102f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:34.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.954+0000 7fb73deed000 1 --2- 172.21.15.107:0/3738532173 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559db4dda040 0x559db4e0fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:34.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.954+0000 7fb7137fe640 0 client.4656 ms_handle_reset on v2:172.21.15.195:6834/3769196693 2023-12-04T05:40:34.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.954+0000 7fb73deed000 1 -- 172.21.15.107:0/3738532173 >> 172.21.15.107:0/3738532173 conn(0x559db4e11840 msgr2=0x559db4e14870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:34.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.954+0000 7fb73deed000 1 -- 172.21.15.107:0/3738532173 shutdown_connections 2023-12-04T05:40:34.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.954+0000 7fb73deed000 1 -- 172.21.15.107:0/3738532173 wait complete. 2023-12-04T05:40:34.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.954+0000 7fb73deed000 20 client.4656 trim_cache size 0 max 16384 2023-12-04T05:40:34.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.974+0000 7fb73deed000 1 Processor -- start 2023-12-04T05:40:34.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.974+0000 7fb73deed000 1 -- start start 2023-12-04T05:40:34.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb73deed000 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559db4dda040 0x559db4dd5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:34.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb73deed000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda960 0x559db4dd6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:34.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb73deed000 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559db4e14fc0 0x559db4e754d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:34.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb73deed000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x559db48dd940 con 0x559db4dda960 2023-12-04T05:40:34.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb73deed000 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x559db4c27320 con 0x559db4dda040 2023-12-04T05:40:34.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb73deed000 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x559db491d1b0 con 0x559db4e14fc0 2023-12-04T05:40:34.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb733fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda960 0x559db4dd6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:34.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb733fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda960 0x559db4dd6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56450/0 (socket says 172.21.15.107:56450) 2023-12-04T05:40:34.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb733fff640 1 -- 172.21.15.107:0/3923292533 learned_addr learned my addr 172.21.15.107:0/3923292533 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:34.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb738b57640 1 --2- 172.21.15.107:0/3923292533 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559db4dda040 0x559db4dd5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:34.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb739358640 1 --2- 172.21.15.107:0/3923292533 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559db4e14fc0 0x559db4e754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:34.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb733fff640 1 -- 172.21.15.107:0/3923292533 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559db4e14fc0 msgr2=0x559db4e754d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:34.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb733fff640 1 --2- 172.21.15.107:0/3923292533 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559db4e14fc0 0x559db4e754d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:34.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb733fff640 1 -- 172.21.15.107:0/3923292533 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559db4dda040 msgr2=0x559db4dd5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:34.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb733fff640 1 --2- 172.21.15.107:0/3923292533 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559db4dda040 0x559db4dd5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:34.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb733fff640 1 -- 172.21.15.107:0/3923292533 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559db4bcb490 con 0x559db4dda960 2023-12-04T05:40:34.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb733fff640 1 --2- 172.21.15.107:0/3923292533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda960 0x559db4dd6480 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb728008010 tx=0x7fb7280b9050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e95fcab9fa718949 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:34.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb71affd640 1 -- 172.21.15.107:0/3923292533 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72801a7f0 con 0x559db4dda960 2023-12-04T05:40:34.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb71affd640 1 -- 172.21.15.107:0/3923292533 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7280ac6e0 con 0x559db4dda960 2023-12-04T05:40:34.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb71affd640 1 -- 172.21.15.107:0/3923292533 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7280abd10 con 0x559db4dda960 2023-12-04T05:40:34.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb73deed000 1 -- 172.21.15.107:0/3923292533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda960 msgr2=0x559db4dd6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:34.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb73deed000 1 --2- 172.21.15.107:0/3923292533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda960 0x559db4dd6480 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb728008010 tx=0x7fb7280b9050 comp rx=0 tx=0).stop 2023-12-04T05:40:34.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb73deed000 1 -- 172.21.15.107:0/3923292533 shutdown_connections 2023-12-04T05:40:34.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb73deed000 1 --2- 172.21.15.107:0/3923292533 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559db4e14fc0 0x559db4e754d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:34.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb73deed000 1 --2- 172.21.15.107:0/3923292533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda960 0x559db4dd6480 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:34.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb73deed000 1 --2- 172.21.15.107:0/3923292533 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559db4dda040 0x559db4dd5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:34.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb73deed000 1 -- 172.21.15.107:0/3923292533 >> 172.21.15.107:0/3923292533 conn(0x559db4e11840 msgr2=0x559db4dd2730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:34.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb73deed000 1 -- 172.21.15.107:0/3923292533 shutdown_connections 2023-12-04T05:40:34.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb73deed000 1 -- 172.21.15.107:0/3923292533 wait complete. 2023-12-04T05:40:34.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb73deed000 1 -- start start 2023-12-04T05:40:34.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.978+0000 7fb73deed000 1 Processor -- start 2023-12-04T05:40:34.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb73deed000 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559db4dda040 0x559db4e2fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:34.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb73deed000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda960 0x559db4e3bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:34.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb73deed000 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559db4e14fc0 0x559db4e3c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:34.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb73deed000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x559db4c20130 con 0x559db4dda960 2023-12-04T05:40:34.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb73deed000 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x559db4c23650 con 0x559db4e14fc0 2023-12-04T05:40:34.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb73deed000 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x559db4dd9bf0 con 0x559db4dda040 2023-12-04T05:40:34.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb73deed000 10 client.0 _ll_register_callbacks cb 0x559db4ba86e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-04T05:40:34.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb733fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda960 0x559db4e3bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:34.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb733fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda960 0x559db4e3bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56452/0 (socket says 172.21.15.107:56452) 2023-12-04T05:40:34.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb733fff640 1 -- 172.21.15.107:0/276495391 learned_addr learned my addr 172.21.15.107:0/276495391 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:34.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb739358640 1 --2- 172.21.15.107:0/276495391 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559db4e14fc0 0x559db4e3c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:34.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb738b57640 1 --2- 172.21.15.107:0/276495391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559db4dda040 0x559db4e2fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:34.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb733fff640 1 -- 172.21.15.107:0/276495391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559db4dda040 msgr2=0x559db4e2fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:34.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb733fff640 1 --2- 172.21.15.107:0/276495391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559db4dda040 0x559db4e2fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:34.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb733fff640 1 -- 172.21.15.107:0/276495391 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559db4e14fc0 msgr2=0x559db4e3c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:34.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb733fff640 1 --2- 172.21.15.107:0/276495391 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559db4e14fc0 0x559db4e3c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:34.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb733fff640 1 -- 172.21.15.107:0/276495391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559db4be5190 con 0x559db4dda960 2023-12-04T05:40:34.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb733fff640 1 --2- 172.21.15.107:0/276495391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda960 0x559db4e3bf30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb728097e80 tx=0x7fb7280a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3be75951b3958873 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:34.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb7137fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-12-04T05:40:34.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb7137fe640 1 -- 172.21.15.107:0/276495391 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7280ab5b0 con 0x559db4dda960 2023-12-04T05:40:34.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb7137fe640 1 -- 172.21.15.107:0/276495391 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb7280ab710 con 0x559db4dda960 2023-12-04T05:40:34.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb73deed000 1 -- 172.21.15.107:0/276495391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559db4c23110 con 0x559db4dda960 2023-12-04T05:40:34.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb7137fe640 1 -- 172.21.15.107:0/276495391 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7280ab8d0 con 0x559db4dda960 2023-12-04T05:40:34.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb73deed000 10 client.4662 fetch_fsmap learned FSMap version 17 2023-12-04T05:40:34.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb73deed000 1 -- 172.21.15.107:0/276495391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x559db491ccc0 con 0x559db4dda960 2023-12-04T05:40:34.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb7137fe640 1 -- 172.21.15.107:0/276495391 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559db4c23110 con 0x559db4dda960 2023-12-04T05:40:34.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb7137fe640 1 -- 172.21.15.107:0/276495391 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 17) v1 ==== 4492+0+0 (secure 0 0 0) 0x7fb7280ca070 con 0x559db4dda960 2023-12-04T05:40:34.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb73deed000 10 client.4662 fetch_fsmap finished waiting for FSMap version 17 2023-12-04T05:40:34.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb73deed000 10 client.4662 resolve_mds: resolved name 'b' to daemon mds.b 2023-12-04T05:40:34.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb73deed000 20 client.4662 populate_metadata read hostname 'smithi107' 2023-12-04T05:40:34.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb73deed000 1 --2- 172.21.15.107:0/276495391 >> [v2:172.21.15.195:6834/3769196693,v1:172.21.15.195:6836/3769196693] conn(0x559db4e7b320 0x559db4e7b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:34.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb73deed000 4 client.4662 mds_command: new command op to 4250 tid=0 multi_id=0 [{"prefix": "status"}] 2023-12-04T05:40:34.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb73deed000 1 -- 172.21.15.107:0/276495391 --> [v2:172.21.15.195:6834/3769196693,v1:172.21.15.195:6836/3769196693] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559db4c51650 con 0x559db4e7b320 2023-12-04T05:40:34.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb738b57640 1 --2- 172.21.15.107:0/276495391 >> [v2:172.21.15.195:6834/3769196693,v1:172.21.15.195:6836/3769196693] conn(0x559db4e7b320 0x559db4e7b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:34.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb738b57640 1 -- 172.21.15.107:0/276495391 >> [v2:172.21.15.195:6834/3769196693,v1:172.21.15.195:6836/3769196693] conn(0x559db4e7b320 msgr2=0x559db4e7b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2023-12-04T05:40:34.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb738b57640 1 -- 172.21.15.107:0/276495391 >> [v2:172.21.15.195:6834/3769196693,v1:172.21.15.195:6836/3769196693] conn(0x559db4e7b320 msgr2=0x559db4e7b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T05:40:34.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb738b57640 1 --2- 172.21.15.107:0/276495391 >> [v2:172.21.15.195:6834/3769196693,v1:172.21.15.195:6836/3769196693] conn(0x559db4e7b320 0x559db4e7b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T05:40:34.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:34.982+0000 7fb738b57640 1 --2- 172.21.15.107:0/276495391 >> [v2:172.21.15.195:6834/3769196693,v1:172.21.15.195:6836/3769196693] conn(0x559db4e7b320 0x559db4e7b700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:40:35.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb738b57640 1 --2- 172.21.15.107:0/276495391 >> [v2:172.21.15.195:6834/3769196693,v1:172.21.15.195:6836/3769196693] conn(0x559db4e7b320 0x559db4e7b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb738b57640 1 --2- 172.21.15.107:0/276495391 >> [v2:172.21.15.195:6834/3769196693,v1:172.21.15.195:6836/3769196693] conn(0x559db4e7b320 0x559db4e7b700 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1f256c1ba723f5b2 server_cookie=7ebbc774b95a5110 in_seq=0 out_seq=0 2023-12-04T05:40:35.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb7137fe640 10 client.4662 ms_handle_connect on v2:172.21.15.195:6834/3769196693 2023-12-04T05:40:35.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb7137fe640 1 -- 172.21.15.107:0/276495391 <== mds.4 v2:172.21.15.195:6834/3769196693 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x559db4c51650 con 0x559db4e7b320 2023-12-04T05:40:35.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb7137fe640 10 client.4662 handle_command_reply: tid=0 2023-12-04T05:40:35.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb73deed000 1 client.4662 shutdown 2023-12-04T05:40:35.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb73deed000 1 -- 172.21.15.107:0/276495391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda960 msgr2=0x559db4e3bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb73deed000 1 --2- 172.21.15.107:0/276495391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda960 0x559db4e3bf30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb728097e80 tx=0x7fb7280a7000 comp rx=0 tx=0).stop 2023-12-04T05:40:35.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb73deed000 1 -- 172.21.15.107:0/276495391 shutdown_connections 2023-12-04T05:40:35.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb73deed000 1 --2- 172.21.15.107:0/276495391 >> [v2:172.21.15.195:6834/3769196693,v1:172.21.15.195:6836/3769196693] conn(0x559db4e7b320 0x559db4e7b700 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb73deed000 1 --2- 172.21.15.107:0/276495391 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559db4e14fc0 0x559db4e3c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb7137fe640 0 client.4662 ms_handle_reset on v2:172.21.15.195:6834/3769196693 2023-12-04T05:40:35.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb73deed000 1 --2- 172.21.15.107:0/276495391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559db4dda960 0x559db4e3bf30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb73deed000 1 --2- 172.21.15.107:0/276495391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559db4dda040 0x559db4e2fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb73deed000 1 -- 172.21.15.107:0/276495391 >> 172.21.15.107:0/276495391 conn(0x559db4e11840 msgr2=0x559db4dd2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb73deed000 1 -- 172.21.15.107:0/276495391 shutdown_connections 2023-12-04T05:40:35.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb73deed000 1 -- 172.21.15.107:0/276495391 wait complete. 2023-12-04T05:40:35.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb73deed000 20 client.4662 trim_cache size 0 max 16384 2023-12-04T05:40:35.192 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-04T05:40:35.192 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "603e856b-31ac-475a-8f35-21ab7faadd48", 2023-12-04T05:40:35.192 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 4, 2023-12-04T05:40:35.192 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 4250, 2023-12-04T05:40:35.192 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-12-04T05:40:35.192 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-12-04T05:40:35.192 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2023-12-04T05:40:35.192 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 11.268874610999999, 2023-12-04T05:40:35.192 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 17, 2023-12-04T05:40:35.192 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 21, 2023-12-04T05:40:35.192 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 21, 2023-12-04T05:40:35.193 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 41.939314912999997 2023-12-04T05:40:35.193 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-04T05:40:35.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb73adbb640 1 -- 172.21.15.107:0/3873061897 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb720066900 msgr2=0x7fb720068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb73adbb640 1 --2- 172.21.15.107:0/3873061897 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb720066900 0x7fb720068dc0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb7280ae000 tx=0x7fb728008040 comp rx=0 tx=0).stop 2023-12-04T05:40:35.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb73adbb640 1 -- 172.21.15.107:0/3873061897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb734072110 msgr2=0x7fb7341f4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb73adbb640 1 --2- 172.21.15.107:0/3873061897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb734072110 0x7fb7341f4a70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb72400c7f0 tx=0x7fb72400ccb0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb73adbb640 1 -- 172.21.15.107:0/3873061897 shutdown_connections 2023-12-04T05:40:35.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb73adbb640 1 --2- 172.21.15.107:0/3873061897 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb720066900 0x7fb720068dc0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb73adbb640 1 --2- 172.21.15.107:0/3873061897 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb7340c56f0 0x7fb7341367a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb73adbb640 1 --2- 172.21.15.107:0/3873061897 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb7340724f0 0x7fb73412ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb73adbb640 1 --2- 172.21.15.107:0/3873061897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb734072110 0x7fb7341f4a70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.186+0000 7fb73adbb640 1 -- 172.21.15.107:0/3873061897 >> 172.21.15.107:0/3873061897 conn(0x7fb7340c7ff0 msgr2=0x7fb73406d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.190+0000 7fb73adbb640 1 -- 172.21.15.107:0/3873061897 shutdown_connections 2023-12-04T05:40:35.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.190+0000 7fb73adbb640 1 -- 172.21.15.107:0/3873061897 wait complete. 2023-12-04T05:40:35.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2023-12-04T05:40:35.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9f1a18640 1 Processor -- start 2023-12-04T05:40:35.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9f1a18640 1 -- start start 2023-12-04T05:40:35.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9f1a18640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd9ec0724f0 0x7fd9ec158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9f1a18640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9ec158da0 con 0x7fd9ec072110 2023-12-04T05:40:35.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9f1a18640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9ec1583d0 con 0x7fd9ec0c56f0 2023-12-04T05:40:35.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9f1a18640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec0c6170 con 0x7fd9ec0724f0 2023-12-04T05:40:35.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9eaffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd9ec072110 0x7fd9ec0c7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42374/0 (socket says 172.21.15.107:42374) 2023-12-04T05:40:35.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9eaffd640 1 -- 172.21.15.107:0/4162918070 learned_addr learned my addr 172.21.15.107:0/4162918070 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:35.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9ea7fc640 1 --2- 172.21.15.107:0/4162918070 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd9ec0724f0 0x7fd9ec158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9e9ffb640 1 -- 172.21.15.107:0/4162918070 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594924112 0 0) 0x7fd9dc001120 con 0x7fd9ec072110 2023-12-04T05:40:35.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9e9ffb640 1 -- 172.21.15.107:0/4162918070 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9d4003530 con 0x7fd9ec072110 2023-12-04T05:40:35.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9e9ffb640 1 -- 172.21.15.107:0/4162918070 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1133676555 0 0) 0x7fd9d4003530 con 0x7fd9ec072110 2023-12-04T05:40:35.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9e9ffb640 1 -- 172.21.15.107:0/4162918070 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd9ec0724f0 msgr2=0x7fd9ec158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9e9ffb640 1 --2- 172.21.15.107:0/4162918070 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd9ec0724f0 0x7fd9ec158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9e9ffb640 1 -- 172.21.15.107:0/4162918070 >> v1:172.21.15.195:6789/0 conn(0x7fd9ec0c56f0 legacy=0x7fd9ec140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9e9ffb640 1 -- 172.21.15.107:0/4162918070 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ec157da0 con 0x7fd9ec072110 2023-12-04T05:40:35.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9e9ffb640 1 -- 172.21.15.107:0/4162918070 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7fd9dc001810 con 0x7fd9ec072110 2023-12-04T05:40:35.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9e9ffb640 1 -- 172.21.15.107:0/4162918070 >> v1:172.21.15.107:6789/0 conn(0x7fd9ec072110 legacy=0x7fd9ec0c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9e9ffb640 1 --2- 172.21.15.107:0/4162918070 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd9ec0724f0 0x7fd9d40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9e9ffb640 1 --2- 172.21.15.107:0/4162918070 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9d4003f30 0x7fd9d40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9e9ffb640 1 --2- 172.21.15.107:0/4162918070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9d4004d10 0x7fd9d4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9e9ffb640 1 -- 172.21.15.107:0/4162918070 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec0c6170 con 0x7fd9d4004d10 2023-12-04T05:40:35.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9e9ffb640 1 -- 172.21.15.107:0/4162918070 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d400bb00 con 0x7fd9d4003f30 2023-12-04T05:40:35.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9e9ffb640 1 -- 172.21.15.107:0/4162918070 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d400bcc0 con 0x7fd9ec0724f0 2023-12-04T05:40:35.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9eb7fe640 1 --2- 172.21.15.107:0/4162918070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9d4004d10 0x7fd9d4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9ea7fc640 1 --2- 172.21.15.107:0/4162918070 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9d4003f30 0x7fd9d40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9eb7fe640 1 -- 172.21.15.107:0/4162918070 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd9ec0724f0 msgr2=0x7fd9d40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9eb7fe640 1 --2- 172.21.15.107:0/4162918070 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd9ec0724f0 0x7fd9d40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9eb7fe640 1 -- 172.21.15.107:0/4162918070 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9d4003f30 msgr2=0x7fd9d40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9eb7fe640 1 --2- 172.21.15.107:0/4162918070 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9d4003f30 0x7fd9d40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.346+0000 7fd9eb7fe640 1 -- 172.21.15.107:0/4162918070 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d4003530 con 0x7fd9d4004d10 2023-12-04T05:40:35.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.350+0000 7fd9eb7fe640 1 --2- 172.21.15.107:0/4162918070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9d4004d10 0x7fd9d4005190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4001e70 tx=0x7fd9e401c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f94a47b1124db60 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.350+0000 7fd9e9ffb640 1 -- 172.21.15.107:0/4162918070 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4003b80 con 0x7fd9d4004d10 2023-12-04T05:40:35.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.350+0000 7fd9e9ffb640 1 -- 172.21.15.107:0/4162918070 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e4003ce0 con 0x7fd9d4004d10 2023-12-04T05:40:35.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.350+0000 7fd9e9ffb640 1 -- 172.21.15.107:0/4162918070 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e40052c0 con 0x7fd9d4004d10 2023-12-04T05:40:35.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.350+0000 7fd9f1a18640 1 -- 172.21.15.107:0/4162918070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9d4004d10 msgr2=0x7fd9d4005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.350+0000 7fd9f1a18640 1 --2- 172.21.15.107:0/4162918070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9d4004d10 0x7fd9d4005190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4001e70 tx=0x7fd9e401c760 comp rx=0 tx=0).stop 2023-12-04T05:40:35.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.350+0000 7fd9eb7fe640 1 -- 172.21.15.107:0/4162918070 reap_dead start 2023-12-04T05:40:35.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.350+0000 7fd9f1a18640 1 -- 172.21.15.107:0/4162918070 shutdown_connections 2023-12-04T05:40:35.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.350+0000 7fd9f1a18640 1 -- 172.21.15.107:0/4162918070 >> 172.21.15.107:0/4162918070 conn(0x7fd9ec0c7ff0 msgr2=0x7fd9ec0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.350+0000 7fd9f1a18640 1 -- 172.21.15.107:0/4162918070 shutdown_connections 2023-12-04T05:40:35.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.350+0000 7fd9f1a18640 1 -- 172.21.15.107:0/4162918070 wait complete. 2023-12-04T05:40:35.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.350+0000 7fd9f1a18640 1 Processor -- start 2023-12-04T05:40:35.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.350+0000 7fd9f1a18640 1 -- start start 2023-12-04T05:40:35.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.350+0000 7fd9f1a18640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd9d4004d10 0x7fd9ec158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.350+0000 7fd9f1a18640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9ec071810 0x7fd9ec156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9f1a18640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec1536e0 0x7fd9ec1f2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9f1a18640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec1efe40 con 0x7fd9ec1536e0 2023-12-04T05:40:35.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9f1a18640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec159780 con 0x7fd9ec071810 2023-12-04T05:40:35.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9f1a18640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec159a40 con 0x7fd9d4004d10 2023-12-04T05:40:35.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9eb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec1536e0 0x7fd9ec1f2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9eb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec1536e0 0x7fd9ec1f2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56464/0 (socket says 172.21.15.107:56464) 2023-12-04T05:40:35.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9eb7fe640 1 -- 172.21.15.107:0/1044991364 learned_addr learned my addr 172.21.15.107:0/1044991364 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9eaffd640 1 --2- 172.21.15.107:0/1044991364 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd9d4004d10 0x7fd9ec158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9ea7fc640 1 --2- 172.21.15.107:0/1044991364 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9ec071810 0x7fd9ec156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9eb7fe640 1 -- 172.21.15.107:0/1044991364 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd9d4004d10 msgr2=0x7fd9ec158f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9eb7fe640 1 --2- 172.21.15.107:0/1044991364 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd9d4004d10 0x7fd9ec158f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9eb7fe640 1 -- 172.21.15.107:0/1044991364 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9ec071810 msgr2=0x7fd9ec156c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9eb7fe640 1 --2- 172.21.15.107:0/1044991364 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9ec071810 0x7fd9ec156c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9eb7fe640 1 -- 172.21.15.107:0/1044991364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ec1ef7f0 con 0x7fd9ec1536e0 2023-12-04T05:40:35.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9ea7fc640 1 --2- 172.21.15.107:0/1044991364 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9ec071810 0x7fd9ec156c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:35.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9eb7fe640 1 --2- 172.21.15.107:0/1044991364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec1536e0 0x7fd9ec1f2a90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd9e401cc40 tx=0x7fd9d4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9d3fff640 1 -- 172.21.15.107:0/1044991364 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4006d60 con 0x7fd9ec1536e0 2023-12-04T05:40:35.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9d3fff640 1 -- 172.21.15.107:0/1044991364 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e4007dc0 con 0x7fd9ec1536e0 2023-12-04T05:40:35.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9f1a18640 1 -- 172.21.15.107:0/1044991364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9ec1efad0 con 0x7fd9ec1536e0 2023-12-04T05:40:35.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9d3fff640 1 -- 172.21.15.107:0/1044991364 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4007230 con 0x7fd9ec1536e0 2023-12-04T05:40:35.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9f1a18640 1 -- 172.21.15.107:0/1044991364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ec1ef4a0 con 0x7fd9ec1536e0 2023-12-04T05:40:35.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9d3fff640 1 -- 172.21.15.107:0/1044991364 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fd9e4007390 con 0x7fd9ec1536e0 2023-12-04T05:40:35.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9d3fff640 1 --2- 172.21.15.107:0/1044991364 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fd9d8066900 0x7fd9d8068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9d3fff640 1 -- 172.21.15.107:0/1044991364 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9e411e690 con 0x7fd9ec1536e0 2023-12-04T05:40:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9eaffd640 1 --2- 172.21.15.107:0/1044991364 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fd9d8066900 0x7fd9d8068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9eaffd640 1 --2- 172.21.15.107:0/1044991364 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fd9d8066900 0x7fd9d8068dc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc001ec0 tx=0x7fd9dc0016f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9f4b4a000 1 Processor -- start 2023-12-04T05:40:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.354+0000 7fd9f4b4a000 1 -- start start 2023-12-04T05:40:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421e57040 0x559421e57420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559421e57960 0x559421e91a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559421e91fc0 0x559421e923a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x559421d18110 con 0x559421e57040 2023-12-04T05:40:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x559421ca89a0 con 0x559421e91fc0 2023-12-04T05:40:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x559421ca5ed0 con 0x559421e57960 2023-12-04T05:40:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9eaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421e57040 0x559421e57420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9eaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421e57040 0x559421e57420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56466/0 (socket says 172.21.15.107:56466) 2023-12-04T05:40:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9eaffd640 1 -- 172.21.15.107:0/4114012083 learned_addr learned my addr 172.21.15.107:0/4114012083 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9eb7fe640 1 --2- 172.21.15.107:0/4114012083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559421e91fc0 0x559421e923a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9eaffd640 1 -- 172.21.15.107:0/4114012083 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559421e57960 msgr2=0x559421e91a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9ea7fc640 1 --2- 172.21.15.107:0/4114012083 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559421e57960 0x559421e91a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9eaffd640 1 --2- 172.21.15.107:0/4114012083 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559421e57960 0x559421e91a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9eaffd640 1 -- 172.21.15.107:0/4114012083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559421e91fc0 msgr2=0x559421e923a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9eaffd640 1 --2- 172.21.15.107:0/4114012083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559421e91fc0 0x559421e923a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9eaffd640 1 -- 172.21.15.107:0/4114012083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559421999890 con 0x559421e57040 2023-12-04T05:40:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9eb7fe640 1 --2- 172.21.15.107:0/4114012083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559421e91fc0 0x559421e923a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9eaffd640 1 --2- 172.21.15.107:0/4114012083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421e57040 0x559421e57420 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00eb50 tx=0x7fd9dc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d77f061ecadd0f7f server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9d17fa640 1 -- 172.21.15.107:0/4114012083 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc004e60 con 0x559421e57040 2023-12-04T05:40:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9d17fa640 1 -- 172.21.15.107:0/4114012083 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9dc005260 con 0x559421e57040 2023-12-04T05:40:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9d17fa640 1 -- 172.21.15.107:0/4114012083 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc0055d0 con 0x559421e57040 2023-12-04T05:40:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/4114012083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421e57040 msgr2=0x559421e57420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 --2- 172.21.15.107:0/4114012083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421e57040 0x559421e57420 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00eb50 tx=0x7fd9dc005f90 comp rx=0 tx=0).stop 2023-12-04T05:40:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/4114012083 shutdown_connections 2023-12-04T05:40:35.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 --2- 172.21.15.107:0/4114012083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559421e91fc0 0x559421e923a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 --2- 172.21.15.107:0/4114012083 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559421e57960 0x559421e91a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 --2- 172.21.15.107:0/4114012083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421e57040 0x559421e57420 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/4114012083 >> 172.21.15.107:0/4114012083 conn(0x559421e8e840 msgr2=0x559421e90c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/4114012083 shutdown_connections 2023-12-04T05:40:35.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/4114012083 wait complete. 2023-12-04T05:40:35.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 -- start start 2023-12-04T05:40:35.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 Processor -- start 2023-12-04T05:40:35.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559421e57040 0x559421e8cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421e57960 0x559421e8d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559421e91fc0 0x559421e53400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x559421ca17c0 con 0x559421e57960 2023-12-04T05:40:35.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x559421c59bf0 con 0x559421e57040 2023-12-04T05:40:35.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x559421c5fa90 con 0x559421e91fc0 2023-12-04T05:40:35.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9f4b4a000 10 client.0 _ll_register_callbacks cb 0x559421e3ce20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-04T05:40:35.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9ea7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421e57960 0x559421e8d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9ea7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421e57960 0x559421e8d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56474/0 (socket says 172.21.15.107:56474) 2023-12-04T05:40:35.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9ea7fc640 1 -- 172.21.15.107:0/3102324041 learned_addr learned my addr 172.21.15.107:0/3102324041 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9eaffd640 1 --2- 172.21.15.107:0/3102324041 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559421e57040 0x559421e8cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9eb7fe640 1 --2- 172.21.15.107:0/3102324041 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559421e91fc0 0x559421e53400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9ea7fc640 1 -- 172.21.15.107:0/3102324041 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559421e91fc0 msgr2=0x559421e53400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9ea7fc640 1 --2- 172.21.15.107:0/3102324041 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559421e91fc0 0x559421e53400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9ea7fc640 1 -- 172.21.15.107:0/3102324041 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559421e57040 msgr2=0x559421e8cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9ea7fc640 1 --2- 172.21.15.107:0/3102324041 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559421e57040 0x559421e8cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.358+0000 7fd9ea7fc640 1 -- 172.21.15.107:0/3102324041 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9dc00f040 con 0x559421e57960 2023-12-04T05:40:35.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9ea7fc640 1 --2- 172.21.15.107:0/3102324041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421e57960 0x559421e8d2f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd9e00135f0 tx=0x7fd9e0013ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=941846f3bb55c1fc server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9ca7fc640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-12-04T05:40:35.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9ca7fc640 1 -- 172.21.15.107:0/3102324041 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e00158f0 con 0x559421e57960 2023-12-04T05:40:35.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9ca7fc640 1 -- 172.21.15.107:0/3102324041 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9e0019070 con 0x559421e57960 2023-12-04T05:40:35.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/3102324041 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559421c63470 con 0x559421e57960 2023-12-04T05:40:35.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9ca7fc640 1 -- 172.21.15.107:0/3102324041 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9e0015a50 con 0x559421e57960 2023-12-04T05:40:35.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9f4b4a000 10 client.4674 fetch_fsmap learned FSMap version 17 2023-12-04T05:40:35.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/3102324041 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x559421c62190 con 0x559421e57960 2023-12-04T05:40:35.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9ca7fc640 1 -- 172.21.15.107:0/3102324041 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559421c63470 con 0x559421e57960 2023-12-04T05:40:35.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9ca7fc640 1 -- 172.21.15.107:0/3102324041 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 17) v1 ==== 4492+0+0 (secure 0 0 0) 0x7fd9e0015bb0 con 0x559421e57960 2023-12-04T05:40:35.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9f4b4a000 10 client.4674 fetch_fsmap finished waiting for FSMap version 17 2023-12-04T05:40:35.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9f4b4a000 10 client.4674 resolve_mds: resolved name 'e' to daemon mds.e 2023-12-04T05:40:35.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9f4b4a000 20 client.4674 populate_metadata read hostname 'smithi107' 2023-12-04T05:40:35.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9f4b4a000 1 --2- 172.21.15.107:0/3102324041 >> [v2:172.21.15.107:6834/3444851515,v1:172.21.15.107:6836/3444851515] conn(0x559421e541a0 0x559421e54580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9f4b4a000 4 client.4674 mds_command: new command op to 4260 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-12-04T05:40:35.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/3102324041 --> [v2:172.21.15.107:6834/3444851515,v1:172.21.15.107:6836/3444851515] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559421c66b50 con 0x559421e541a0 2023-12-04T05:40:35.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9eaffd640 1 --2- 172.21.15.107:0/3102324041 >> [v2:172.21.15.107:6834/3444851515,v1:172.21.15.107:6836/3444851515] conn(0x559421e541a0 0x559421e54580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9ca7fc640 10 client.4674 ms_handle_connect on v2:172.21.15.107:6834/3444851515 2023-12-04T05:40:35.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9eaffd640 1 --2- 172.21.15.107:0/3102324041 >> [v2:172.21.15.107:6834/3444851515,v1:172.21.15.107:6836/3444851515] conn(0x559421e541a0 0x559421e54580 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=895ce3b6bd628eb2 server_cookie=c020c7ef4fcc6da6 in_seq=0 out_seq=0 2023-12-04T05:40:35.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9ca7fc640 1 -- 172.21.15.107:0/3102324041 <== mds.3 v2:172.21.15.107:6834/3444851515 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559421c66b50 con 0x559421e541a0 2023-12-04T05:40:35.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9ca7fc640 10 client.4674 handle_command_reply: tid=0 2023-12-04T05:40:35.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9f4b4a000 1 client.4674 shutdown 2023-12-04T05:40:35.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/3102324041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421e57960 msgr2=0x559421e8d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9f4b4a000 1 --2- 172.21.15.107:0/3102324041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421e57960 0x559421e8d2f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd9e00135f0 tx=0x7fd9e0013ab0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/3102324041 shutdown_connections 2023-12-04T05:40:35.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9f4b4a000 1 --2- 172.21.15.107:0/3102324041 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559421e91fc0 0x559421e53400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9f4b4a000 1 --2- 172.21.15.107:0/3102324041 >> [v2:172.21.15.107:6834/3444851515,v1:172.21.15.107:6836/3444851515] conn(0x559421e541a0 0x559421e54580 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9f4b4a000 1 --2- 172.21.15.107:0/3102324041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421e57960 0x559421e8d2f0 secure :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd9e00135f0 tx=0x7fd9e0013ab0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9f4b4a000 1 --2- 172.21.15.107:0/3102324041 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559421e57040 0x559421e8cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/3102324041 >> 172.21.15.107:0/3102324041 conn(0x559421e8e840 msgr2=0x559421e91870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9ca7fc640 0 client.4674 ms_handle_reset on v2:172.21.15.107:6834/3444851515 2023-12-04T05:40:35.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/3102324041 shutdown_connections 2023-12-04T05:40:35.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/3102324041 wait complete. 2023-12-04T05:40:35.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.362+0000 7fd9f4b4a000 20 client.4674 trim_cache size 0 max 16384 2023-12-04T05:40:35.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.378+0000 7fd9f4b4a000 1 Processor -- start 2023-12-04T05:40:35.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.378+0000 7fd9f4b4a000 1 -- start start 2023-12-04T05:40:35.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.378+0000 7fd9f4b4a000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421e57040 0x559421eeded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.378+0000 7fd9f4b4a000 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559421e91fc0 0x559421eee410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.378+0000 7fd9f4b4a000 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559421eee950 0x559421e51fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.378+0000 7fd9f4b4a000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55942195a940 con 0x559421e57040 2023-12-04T05:40:35.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.378+0000 7fd9f4b4a000 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x559421c9bac0 con 0x559421e91fc0 2023-12-04T05:40:35.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.378+0000 7fd9f4b4a000 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x55942199a1b0 con 0x559421eee950 2023-12-04T05:40:35.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.378+0000 7fd9eaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421e57040 0x559421eeded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.378+0000 7fd9eaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421e57040 0x559421eeded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56484/0 (socket says 172.21.15.107:56484) 2023-12-04T05:40:35.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.378+0000 7fd9eaffd640 1 -- 172.21.15.107:0/3191028867 learned_addr learned my addr 172.21.15.107:0/3191028867 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.378+0000 7fd9eaffd640 1 -- 172.21.15.107:0/3191028867 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559421eee950 msgr2=0x559421e51fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.378+0000 7fd9eaffd640 1 --2- 172.21.15.107:0/3191028867 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559421eee950 0x559421e51fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.378+0000 7fd9eaffd640 1 -- 172.21.15.107:0/3191028867 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559421e91fc0 msgr2=0x559421eee410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.378+0000 7fd9eaffd640 1 --2- 172.21.15.107:0/3191028867 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559421e91fc0 0x559421eee410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.378+0000 7fd9eaffd640 1 -- 172.21.15.107:0/3191028867 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559421c48490 con 0x559421e57040 2023-12-04T05:40:35.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.378+0000 7fd9eaffd640 1 --2- 172.21.15.107:0/3191028867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421e57040 0x559421eeded0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc005fd0 tx=0x7fd9dc013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6edfa3ca7db5c0f server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.378+0000 7fd9d1ffb640 1 -- 172.21.15.107:0/3191028867 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc026040 con 0x559421e57040 2023-12-04T05:40:35.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.378+0000 7fd9d1ffb640 1 -- 172.21.15.107:0/3191028867 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9dc005500 con 0x559421e57040 2023-12-04T05:40:35.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.378+0000 7fd9d1ffb640 1 -- 172.21.15.107:0/3191028867 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc02c420 con 0x559421e57040 2023-12-04T05:40:35.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.378+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/3191028867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421e57040 msgr2=0x559421eeded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.378+0000 7fd9f4b4a000 1 --2- 172.21.15.107:0/3191028867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421e57040 0x559421eeded0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc005fd0 tx=0x7fd9dc013000 comp rx=0 tx=0).stop 2023-12-04T05:40:35.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/3191028867 shutdown_connections 2023-12-04T05:40:35.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 --2- 172.21.15.107:0/3191028867 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559421eee950 0x559421e51fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 --2- 172.21.15.107:0/3191028867 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559421e91fc0 0x559421eee410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 --2- 172.21.15.107:0/3191028867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421e57040 0x559421eeded0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/3191028867 >> 172.21.15.107:0/3191028867 conn(0x559421e8e840 msgr2=0x559421e4f730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/3191028867 shutdown_connections 2023-12-04T05:40:35.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/3191028867 wait complete. 2023-12-04T05:40:35.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 -- start start 2023-12-04T05:40:35.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 Processor -- start 2023-12-04T05:40:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559421e57040 0x559421eac140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559421e91fc0 0x559421eac680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421eee950 0x559421eacbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x559421d28570 con 0x559421eee950 2023-12-04T05:40:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x559421ca4320 con 0x559421e91fc0 2023-12-04T05:40:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x559421c9efd0 con 0x559421e57040 2023-12-04T05:40:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 10 client.0 _ll_register_callbacks cb 0x559421c256e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-04T05:40:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9eb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421eee950 0x559421eacbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9eb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421eee950 0x559421eacbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56494/0 (socket says 172.21.15.107:56494) 2023-12-04T05:40:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9eb7fe640 1 -- 172.21.15.107:0/248942682 learned_addr learned my addr 172.21.15.107:0/248942682 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9eb7fe640 1 -- 172.21.15.107:0/248942682 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559421e57040 msgr2=0x559421eac140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9eb7fe640 1 --2- 172.21.15.107:0/248942682 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559421e57040 0x559421eac140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9eb7fe640 1 -- 172.21.15.107:0/248942682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559421e91fc0 msgr2=0x559421eac680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9ea7fc640 1 --2- 172.21.15.107:0/248942682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559421e91fc0 0x559421eac680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9eb7fe640 1 --2- 172.21.15.107:0/248942682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559421e91fc0 0x559421eac680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9eb7fe640 1 -- 172.21.15.107:0/248942682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559421999890 con 0x559421eee950 2023-12-04T05:40:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9eb7fe640 1 --2- 172.21.15.107:0/248942682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421eee950 0x559421eacbc0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4098170 tx=0x7fd9e40eef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=531ed9325010fbf5 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9ca7fc640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-12-04T05:40:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9ca7fc640 1 -- 172.21.15.107:0/248942682 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e40f3650 con 0x559421eee950 2023-12-04T05:40:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9ca7fc640 1 -- 172.21.15.107:0/248942682 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9e40f7890 con 0x559421eee950 2023-12-04T05:40:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/248942682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559421ca0110 con 0x559421eee950 2023-12-04T05:40:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9ca7fc640 1 -- 172.21.15.107:0/248942682 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9e40ee9f0 con 0x559421eee950 2023-12-04T05:40:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 10 client.4680 fetch_fsmap learned FSMap version 17 2023-12-04T05:40:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/248942682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x559421999cc0 con 0x559421eee950 2023-12-04T05:40:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9ca7fc640 1 -- 172.21.15.107:0/248942682 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559421ca0110 con 0x559421eee950 2023-12-04T05:40:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9ca7fc640 1 -- 172.21.15.107:0/248942682 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 17) v1 ==== 4492+0+0 (secure 0 0 0) 0x7fd9e40f7400 con 0x559421eee950 2023-12-04T05:40:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 10 client.4680 fetch_fsmap finished waiting for FSMap version 17 2023-12-04T05:40:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 10 client.4680 resolve_mds: resolved name 'e' to daemon mds.e 2023-12-04T05:40:35.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 20 client.4680 populate_metadata read hostname 'smithi107' 2023-12-04T05:40:35.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 --2- 172.21.15.107:0/248942682 >> [v2:172.21.15.107:6834/3444851515,v1:172.21.15.107:6836/3444851515] conn(0x559421ead580 0x559421f00e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 4 client.4680 mds_command: new command op to 4260 tid=0 multi_id=0 [{"prefix": "status"}] 2023-12-04T05:40:35.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/248942682 --> [v2:172.21.15.107:6834/3444851515,v1:172.21.15.107:6836/3444851515] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559421cce650 con 0x559421ead580 2023-12-04T05:40:35.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9eaffd640 1 --2- 172.21.15.107:0/248942682 >> [v2:172.21.15.107:6834/3444851515,v1:172.21.15.107:6836/3444851515] conn(0x559421ead580 0x559421f00e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9ca7fc640 10 client.4680 ms_handle_connect on v2:172.21.15.107:6834/3444851515 2023-12-04T05:40:35.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9eaffd640 1 --2- 172.21.15.107:0/248942682 >> [v2:172.21.15.107:6834/3444851515,v1:172.21.15.107:6836/3444851515] conn(0x559421ead580 0x559421f00e10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=83fc21bbac2705a0 server_cookie=22b9b61a91deda14 in_seq=0 out_seq=0 2023-12-04T05:40:35.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9ca7fc640 1 -- 172.21.15.107:0/248942682 <== mds.3 v2:172.21.15.107:6834/3444851515 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x559421cce650 con 0x559421ead580 2023-12-04T05:40:35.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9ca7fc640 10 client.4680 handle_command_reply: tid=0 2023-12-04T05:40:35.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 client.4680 shutdown 2023-12-04T05:40:35.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/248942682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421eee950 msgr2=0x559421eacbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 --2- 172.21.15.107:0/248942682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421eee950 0x559421eacbc0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4098170 tx=0x7fd9e40eef20 comp rx=0 tx=0).stop 2023-12-04T05:40:35.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/248942682 shutdown_connections 2023-12-04T05:40:35.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 --2- 172.21.15.107:0/248942682 >> [v2:172.21.15.107:6834/3444851515,v1:172.21.15.107:6836/3444851515] conn(0x559421ead580 0x559421f00e10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 --2- 172.21.15.107:0/248942682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559421eee950 0x559421eacbc0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 --2- 172.21.15.107:0/248942682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559421e91fc0 0x559421eac680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 --2- 172.21.15.107:0/248942682 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559421e57040 0x559421eac140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9ca7fc640 0 client.4680 ms_handle_reset on v2:172.21.15.107:6834/3444851515 2023-12-04T05:40:35.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/248942682 >> 172.21.15.107:0/248942682 conn(0x559421e8e840 msgr2=0x559421e533f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/248942682 shutdown_connections 2023-12-04T05:40:35.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 1 -- 172.21.15.107:0/248942682 wait complete. 2023-12-04T05:40:35.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.382+0000 7fd9f4b4a000 20 client.4680 trim_cache size 0 max 16384 2023-12-04T05:40:35.393 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-04T05:40:35.393 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "603e856b-31ac-475a-8f35-21ab7faadd48", 2023-12-04T05:40:35.393 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 3, 2023-12-04T05:40:35.393 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 4260, 2023-12-04T05:40:35.394 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-12-04T05:40:35.394 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-12-04T05:40:35.394 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2023-12-04T05:40:35.394 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 17.501642672999999, 2023-12-04T05:40:35.394 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 17, 2023-12-04T05:40:35.394 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 21, 2023-12-04T05:40:35.394 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 21, 2023-12-04T05:40:35.394 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 42.147569498999999 2023-12-04T05:40:35.394 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-04T05:40:35.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.386+0000 7fd9f1a18640 1 -- 172.21.15.107:0/1044991364 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fd9d8066900 msgr2=0x7fd9d8068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.386+0000 7fd9f1a18640 1 --2- 172.21.15.107:0/1044991364 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fd9d8066900 0x7fd9d8068dc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc001ec0 tx=0x7fd9dc0016f0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.386+0000 7fd9f1a18640 1 -- 172.21.15.107:0/1044991364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec1536e0 msgr2=0x7fd9ec1f2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.386+0000 7fd9f1a18640 1 --2- 172.21.15.107:0/1044991364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec1536e0 0x7fd9ec1f2a90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd9e401cc40 tx=0x7fd9d4005800 comp rx=0 tx=0).stop 2023-12-04T05:40:35.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.386+0000 7fd9f1a18640 1 -- 172.21.15.107:0/1044991364 shutdown_connections 2023-12-04T05:40:35.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.386+0000 7fd9f1a18640 1 --2- 172.21.15.107:0/1044991364 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fd9d8066900 0x7fd9d8068dc0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.386+0000 7fd9f1a18640 1 --2- 172.21.15.107:0/1044991364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec1536e0 0x7fd9ec1f2a90 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.386+0000 7fd9f1a18640 1 --2- 172.21.15.107:0/1044991364 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9ec071810 0x7fd9ec156c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.386+0000 7fd9f1a18640 1 --2- 172.21.15.107:0/1044991364 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd9d4004d10 0x7fd9ec158f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.386+0000 7fd9f1a18640 1 -- 172.21.15.107:0/1044991364 >> 172.21.15.107:0/1044991364 conn(0x7fd9ec0c7ff0 msgr2=0x7fd9ec0c71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.386+0000 7fd9f1a18640 1 -- 172.21.15.107:0/1044991364 shutdown_connections 2023-12-04T05:40:35.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.386+0000 7fd9f1a18640 1 -- 172.21.15.107:0/1044991364 wait complete. 2023-12-04T05:40:35.405 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-12-04T05:40:35.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.514+0000 7f0843bd4640 1 Processor -- start 2023-12-04T05:40:35.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.514+0000 7f0843bd4640 1 -- start start 2023-12-04T05:40:35.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.514+0000 7f0843bd4640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f083c15e6b0 0x7f083c15ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.514+0000 7f0843bd4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f083c0c8c10 0x7f083c15efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.514+0000 7f0843bd4640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f083c1581d0 0x7f083c158660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.514+0000 7f0843bd4640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f083c0c6290 con 0x7f083c0c8c10 2023-12-04T05:40:35.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.514+0000 7f0843bd4640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f083c158ba0 con 0x7f083c1581d0 2023-12-04T05:40:35.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.514+0000 7f0843bd4640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f083c0c5ef0 con 0x7f083c15e6b0 2023-12-04T05:40:35.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.514+0000 7f084116f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f083c0c8c10 0x7f083c15efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.514+0000 7f084116f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f083c0c8c10 0x7f083c15efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56508/0 (socket says 172.21.15.107:56508) 2023-12-04T05:40:35.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.514+0000 7f084116f640 1 -- 172.21.15.107:0/2639146599 learned_addr learned my addr 172.21.15.107:0/2639146599 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.514+0000 7f0842171640 1 --2- 172.21.15.107:0/2639146599 >> v2:172.21.15.195:3300/0 conn(0x7f083c1581d0 0x7f083c158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.514+0000 7f0841970640 1 --2- 172.21.15.107:0/2639146599 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f083c15e6b0 0x7f083c15ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.514+0000 7f084116f640 1 -- 172.21.15.107:0/2639146599 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f083c15e6b0 msgr2=0x7f083c15ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.514+0000 7f084116f640 1 --2- 172.21.15.107:0/2639146599 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f083c15e6b0 0x7f083c15ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.514+0000 7f084116f640 1 -- 172.21.15.107:0/2639146599 >> v2:172.21.15.195:3300/0 conn(0x7f083c1581d0 msgr2=0x7f083c158660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.514+0000 7f084116f640 1 --2- 172.21.15.107:0/2639146599 >> v2:172.21.15.195:3300/0 conn(0x7f083c1581d0 0x7f083c158660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.514+0000 7f084116f640 1 -- 172.21.15.107:0/2639146599 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083c0c25e0 con 0x7f083c0c8c10 2023-12-04T05:40:35.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.514+0000 7f0841970640 1 --2- 172.21.15.107:0/2639146599 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f083c15e6b0 0x7f083c15ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:35.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.514+0000 7f0842171640 1 --2- 172.21.15.107:0/2639146599 >> v2:172.21.15.195:3300/0 conn(0x7f083c1581d0 0x7f083c158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:35.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.518+0000 7f084116f640 1 --2- 172.21.15.107:0/2639146599 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f083c0c8c10 0x7f083c15efd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f083c06f680 tx=0x7f082c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aac7e40ca77d0d7b server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.518+0000 7f084096e640 1 -- 172.21.15.107:0/2639146599 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c0a89b0 con 0x7f083c0c8c10 2023-12-04T05:40:35.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.518+0000 7f084096e640 1 -- 172.21.15.107:0/2639146599 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082c0a8b10 con 0x7f083c0c8c10 2023-12-04T05:40:35.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.518+0000 7f084096e640 1 -- 172.21.15.107:0/2639146599 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c0a8ea0 con 0x7f083c0c8c10 2023-12-04T05:40:35.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.518+0000 7f0843bd4640 1 -- 172.21.15.107:0/2639146599 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f083c0c8c10 msgr2=0x7f083c15efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.518+0000 7f0843bd4640 1 --2- 172.21.15.107:0/2639146599 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f083c0c8c10 0x7f083c15efd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f083c06f680 tx=0x7f082c01ecf0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.518+0000 7f0843bd4640 1 -- 172.21.15.107:0/2639146599 shutdown_connections 2023-12-04T05:40:35.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.518+0000 7f0843bd4640 1 --2- 172.21.15.107:0/2639146599 >> v2:172.21.15.195:3300/0 conn(0x7f083c1581d0 0x7f083c158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.518+0000 7f0843bd4640 1 --2- 172.21.15.107:0/2639146599 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f083c0c8c10 0x7f083c15efd0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.518+0000 7f0843bd4640 1 --2- 172.21.15.107:0/2639146599 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f083c15e6b0 0x7f083c15ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.518+0000 7f0843bd4640 1 -- 172.21.15.107:0/2639146599 >> 172.21.15.107:0/2639146599 conn(0x7f083c0c7ff0 msgr2=0x7f083c0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.518+0000 7f0843bd4640 1 -- 172.21.15.107:0/2639146599 shutdown_connections 2023-12-04T05:40:35.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.518+0000 7f0843bd4640 1 -- 172.21.15.107:0/2639146599 wait complete. 2023-12-04T05:40:35.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f0843bd4640 1 Processor -- start 2023-12-04T05:40:35.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f0843bd4640 1 -- start start 2023-12-04T05:40:35.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f0843bd4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f083c0c8c10 0x7f083c154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f0843bd4640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f083c1581d0 0x7f083c156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f0843bd4640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f083c15e6b0 0x7f083c156690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f0843bd4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c160ba0 con 0x7f083c0c8c10 2023-12-04T05:40:35.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f0843bd4640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c147020 con 0x7f083c1581d0 2023-12-04T05:40:35.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f0843bd4640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f083c152ed0 con 0x7f083c15e6b0 2023-12-04T05:40:35.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f0841970640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f083c0c8c10 0x7f083c154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f0841970640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f083c0c8c10 0x7f083c154f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56514/0 (socket says 172.21.15.107:56514) 2023-12-04T05:40:35.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f0841970640 1 -- 172.21.15.107:0/3701594430 learned_addr learned my addr 172.21.15.107:0/3701594430 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f084116f640 1 --2- 172.21.15.107:0/3701594430 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f083c1581d0 0x7f083c156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f0842171640 1 --2- 172.21.15.107:0/3701594430 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f083c15e6b0 0x7f083c156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f0841970640 1 -- 172.21.15.107:0/3701594430 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f083c15e6b0 msgr2=0x7f083c156690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f0841970640 1 --2- 172.21.15.107:0/3701594430 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f083c15e6b0 0x7f083c156690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f0841970640 1 -- 172.21.15.107:0/3701594430 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f083c1581d0 msgr2=0x7f083c156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f0841970640 1 --2- 172.21.15.107:0/3701594430 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f083c1581d0 0x7f083c156150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f0841970640 1 -- 172.21.15.107:0/3701594430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082c0094d0 con 0x7f083c0c8c10 2023-12-04T05:40:35.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f084116f640 1 --2- 172.21.15.107:0/3701594430 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f083c1581d0 0x7f083c156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:35.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f0841970640 1 --2- 172.21.15.107:0/3701594430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f083c0c8c10 0x7f083c154f00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f083800c820 tx=0x7f083800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f08327fc640 1 -- 172.21.15.107:0/3701594430 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0838007a30 con 0x7f083c0c8c10 2023-12-04T05:40:35.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f08327fc640 1 -- 172.21.15.107:0/3701594430 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0838007b90 con 0x7f083c0c8c10 2023-12-04T05:40:35.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f0843bd4640 1 -- 172.21.15.107:0/3701594430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f083c155440 con 0x7f083c0c8c10 2023-12-04T05:40:35.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f08327fc640 1 -- 172.21.15.107:0/3701594430 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083800ce10 con 0x7f083c0c8c10 2023-12-04T05:40:35.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f0843bd4640 1 -- 172.21.15.107:0/3701594430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f083c152880 con 0x7f083c0c8c10 2023-12-04T05:40:35.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.522+0000 7f08327fc640 1 -- 172.21.15.107:0/3701594430 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f08380113f0 con 0x7f083c0c8c10 2023-12-04T05:40:35.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f08327fc640 1 --2- 172.21.15.107:0/3701594430 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f08200668d0 0x7f0820068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f08327fc640 1 -- 172.21.15.107:0/3701594430 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0838084620 con 0x7f083c0c8c10 2023-12-04T05:40:35.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f084116f640 1 --2- 172.21.15.107:0/3701594430 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f08200668d0 0x7f0820068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f084116f640 1 --2- 172.21.15.107:0/3701594430 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f08200668d0 0x7f0820068d90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f082c0b2000 tx=0x7f082c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0846d06000 1 Processor -- start 2023-12-04T05:40:35.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0846d06000 1 -- start start 2023-12-04T05:40:35.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0846d06000 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559fb1f39040 0x559fb1f39420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0846d06000 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559fb1f39960 0x559fb1f73a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0846d06000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f73fc0 0x559fb1f743a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0846d06000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fb1dfa110 con 0x559fb1f73fc0 2023-12-04T05:40:35.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0846d06000 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fb1d8a9a0 con 0x559fb1f39040 2023-12-04T05:40:35.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0846d06000 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x559fb1d87ed0 con 0x559fb1f39960 2023-12-04T05:40:35.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0842171640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f73fc0 0x559fb1f743a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0842171640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f73fc0 0x559fb1f743a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56530/0 (socket says 172.21.15.107:56530) 2023-12-04T05:40:35.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0842171640 1 -- 172.21.15.107:0/3344701377 learned_addr learned my addr 172.21.15.107:0/3344701377 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0841970640 1 --2- 172.21.15.107:0/3344701377 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559fb1f39040 0x559fb1f39420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f084116f640 1 --2- 172.21.15.107:0/3344701377 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559fb1f39960 0x559fb1f73a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0842171640 1 -- 172.21.15.107:0/3344701377 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559fb1f39960 msgr2=0x559fb1f73a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0842171640 1 --2- 172.21.15.107:0/3344701377 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559fb1f39960 0x559fb1f73a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0842171640 1 -- 172.21.15.107:0/3344701377 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559fb1f39040 msgr2=0x559fb1f39420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0842171640 1 --2- 172.21.15.107:0/3344701377 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559fb1f39040 0x559fb1f39420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0842171640 1 -- 172.21.15.107:0/3344701377 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559fb1a7b890 con 0x559fb1f73fc0 2023-12-04T05:40:35.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0841970640 1 --2- 172.21.15.107:0/3344701377 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559fb1f39040 0x559fb1f39420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:35.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0842171640 1 --2- 172.21.15.107:0/3344701377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f73fc0 0x559fb1f743a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0834008f80 tx=0x7f083400d8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eed2f4e57c5001f3 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f081f7fe640 1 -- 172.21.15.107:0/3344701377 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083400b6e0 con 0x559fb1f73fc0 2023-12-04T05:40:35.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f081f7fe640 1 -- 172.21.15.107:0/3344701377 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083400b840 con 0x559fb1f73fc0 2023-12-04T05:40:35.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f081f7fe640 1 -- 172.21.15.107:0/3344701377 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083400bbf0 con 0x559fb1f73fc0 2023-12-04T05:40:35.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0846d06000 1 -- 172.21.15.107:0/3344701377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f73fc0 msgr2=0x559fb1f743a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0846d06000 1 --2- 172.21.15.107:0/3344701377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f73fc0 0x559fb1f743a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0834008f80 tx=0x7f083400d8c0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0846d06000 1 -- 172.21.15.107:0/3344701377 shutdown_connections 2023-12-04T05:40:35.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0846d06000 1 --2- 172.21.15.107:0/3344701377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f73fc0 0x559fb1f743a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0846d06000 1 --2- 172.21.15.107:0/3344701377 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559fb1f39960 0x559fb1f73a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0846d06000 1 --2- 172.21.15.107:0/3344701377 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559fb1f39040 0x559fb1f39420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0846d06000 1 -- 172.21.15.107:0/3344701377 >> 172.21.15.107:0/3344701377 conn(0x559fb1f70840 msgr2=0x559fb1f72c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0846d06000 1 -- 172.21.15.107:0/3344701377 shutdown_connections 2023-12-04T05:40:35.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0846d06000 1 -- 172.21.15.107:0/3344701377 wait complete. 2023-12-04T05:40:35.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0846d06000 1 -- start start 2023-12-04T05:40:35.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.526+0000 7f0846d06000 1 Processor -- start 2023-12-04T05:40:35.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559fb1f39040 0x559fb1f6edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f39960 0x559fb1f6f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559fb1f73fc0 0x559fb1f35340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fb1d837c0 con 0x559fb1f39960 2023-12-04T05:40:35.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fb1d3bbf0 con 0x559fb1f39040 2023-12-04T05:40:35.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x559fb1d41a90 con 0x559fb1f73fc0 2023-12-04T05:40:35.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 10 client.0 _ll_register_callbacks cb 0x559fb1f1ee20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-04T05:40:35.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f084116f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f39960 0x559fb1f6f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f084116f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f39960 0x559fb1f6f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56540/0 (socket says 172.21.15.107:56540) 2023-12-04T05:40:35.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f084116f640 1 -- 172.21.15.107:0/2845456733 learned_addr learned my addr 172.21.15.107:0/2845456733 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0841970640 1 --2- 172.21.15.107:0/2845456733 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559fb1f39040 0x559fb1f6edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0842171640 1 --2- 172.21.15.107:0/2845456733 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559fb1f73fc0 0x559fb1f35340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f084116f640 1 -- 172.21.15.107:0/2845456733 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559fb1f73fc0 msgr2=0x559fb1f35340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f084116f640 1 --2- 172.21.15.107:0/2845456733 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559fb1f73fc0 0x559fb1f35340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f084116f640 1 -- 172.21.15.107:0/2845456733 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559fb1f39040 msgr2=0x559fb1f6edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f084116f640 1 --2- 172.21.15.107:0/2845456733 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559fb1f39040 0x559fb1f6edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f084116f640 1 -- 172.21.15.107:0/2845456733 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0834008be0 con 0x559fb1f39960 2023-12-04T05:40:35.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0841970640 1 --2- 172.21.15.107:0/2845456733 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559fb1f39040 0x559fb1f6edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:35.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f084116f640 1 --2- 172.21.15.107:0/2845456733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f39960 0x559fb1f6f2f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f082c0a8ae0 tx=0x7f082c0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3028190c706957cf server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f081cff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-12-04T05:40:35.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f081cff9640 1 -- 172.21.15.107:0/2845456733 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c0c0040 con 0x559fb1f39960 2023-12-04T05:40:35.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f081cff9640 1 -- 172.21.15.107:0/2845456733 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f082c0bb420 con 0x559fb1f39960 2023-12-04T05:40:35.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 1 -- 172.21.15.107:0/2845456733 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559fb1d45470 con 0x559fb1f39960 2023-12-04T05:40:35.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f081cff9640 1 -- 172.21.15.107:0/2845456733 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f082c0bb620 con 0x559fb1f39960 2023-12-04T05:40:35.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 10 client.4692 fetch_fsmap learned FSMap version 17 2023-12-04T05:40:35.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 1 -- 172.21.15.107:0/2845456733 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x559fb1d44190 con 0x559fb1f39960 2023-12-04T05:40:35.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f081cff9640 1 -- 172.21.15.107:0/2845456733 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559fb1d45470 con 0x559fb1f39960 2023-12-04T05:40:35.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f081cff9640 1 -- 172.21.15.107:0/2845456733 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 17) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f082c0bb780 con 0x559fb1f39960 2023-12-04T05:40:35.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 10 client.4692 fetch_fsmap finished waiting for FSMap version 17 2023-12-04T05:40:35.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 10 client.4692 resolve_mds: resolved name 'c' to daemon mds.c 2023-12-04T05:40:35.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 20 client.4692 populate_metadata read hostname 'smithi107' 2023-12-04T05:40:35.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 1 --2- 172.21.15.107:0/2845456733 >> [v2:172.21.15.107:6835/2210527429,v1:172.21.15.107:6837/2210527429] conn(0x559fb1f360b0 0x559fb1f36490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 4 client.4692 mds_command: new command op to 4263 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-12-04T05:40:35.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 1 -- 172.21.15.107:0/2845456733 --> [v2:172.21.15.107:6835/2210527429,v1:172.21.15.107:6837/2210527429] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559fb1d48b50 con 0x559fb1f360b0 2023-12-04T05:40:35.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0841970640 1 --2- 172.21.15.107:0/2845456733 >> [v2:172.21.15.107:6835/2210527429,v1:172.21.15.107:6837/2210527429] conn(0x559fb1f360b0 0x559fb1f36490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0841970640 1 --2- 172.21.15.107:0/2845456733 >> [v2:172.21.15.107:6835/2210527429,v1:172.21.15.107:6837/2210527429] conn(0x559fb1f360b0 0x559fb1f36490 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f73f1d84445d258d server_cookie=de439b0cd7e5ed3c in_seq=0 out_seq=0 2023-12-04T05:40:35.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f081cff9640 10 client.4692 ms_handle_connect on v2:172.21.15.107:6835/2210527429 2023-12-04T05:40:35.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f081cff9640 1 -- 172.21.15.107:0/2845456733 <== mds.2 v2:172.21.15.107:6835/2210527429 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559fb1d48b50 con 0x559fb1f360b0 2023-12-04T05:40:35.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f081cff9640 10 client.4692 handle_command_reply: tid=0 2023-12-04T05:40:35.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 1 client.4692 shutdown 2023-12-04T05:40:35.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 1 -- 172.21.15.107:0/2845456733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f39960 msgr2=0x559fb1f6f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 1 --2- 172.21.15.107:0/2845456733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f39960 0x559fb1f6f2f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f082c0a8ae0 tx=0x7f082c0046d0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 1 -- 172.21.15.107:0/2845456733 shutdown_connections 2023-12-04T05:40:35.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 1 --2- 172.21.15.107:0/2845456733 >> [v2:172.21.15.107:6835/2210527429,v1:172.21.15.107:6837/2210527429] conn(0x559fb1f360b0 0x559fb1f36490 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 1 --2- 172.21.15.107:0/2845456733 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559fb1f73fc0 0x559fb1f35340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 1 --2- 172.21.15.107:0/2845456733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f39960 0x559fb1f6f2f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f081cff9640 0 client.4692 ms_handle_reset on v2:172.21.15.107:6835/2210527429 2023-12-04T05:40:35.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 1 --2- 172.21.15.107:0/2845456733 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559fb1f39040 0x559fb1f6edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 1 -- 172.21.15.107:0/2845456733 >> 172.21.15.107:0/2845456733 conn(0x559fb1f70840 msgr2=0x559fb1f73870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 1 -- 172.21.15.107:0/2845456733 shutdown_connections 2023-12-04T05:40:35.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 1 -- 172.21.15.107:0/2845456733 wait complete. 2023-12-04T05:40:35.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.530+0000 7f0846d06000 20 client.4692 trim_cache size 0 max 16384 2023-12-04T05:40:35.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.546+0000 7f0846d06000 1 Processor -- start 2023-12-04T05:40:35.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.546+0000 7f0846d06000 1 -- start start 2023-12-04T05:40:35.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.546+0000 7f0846d06000 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559fb1f70840 0x559fb1f34f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.546+0000 7f0846d06000 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559fb1f39040 0x559fb1f35450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.546+0000 7f0846d06000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f39960 0x559fb1fd1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.546+0000 7f0846d06000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fb1a3c940 con 0x559fb1f39960 2023-12-04T05:40:35.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.546+0000 7f0846d06000 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fb1d58b70 con 0x559fb1f70840 2023-12-04T05:40:35.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.546+0000 7f0846d06000 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x559fb1a7c1b0 con 0x559fb1f39040 2023-12-04T05:40:35.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.546+0000 7f0842171640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f39960 0x559fb1fd1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.546+0000 7f0842171640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f39960 0x559fb1fd1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56552/0 (socket says 172.21.15.107:56552) 2023-12-04T05:40:35.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.546+0000 7f0842171640 1 -- 172.21.15.107:0/3337465562 learned_addr learned my addr 172.21.15.107:0/3337465562 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0842171640 1 -- 172.21.15.107:0/3337465562 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559fb1f39040 msgr2=0x559fb1f35450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0842171640 1 --2- 172.21.15.107:0/3337465562 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559fb1f39040 0x559fb1f35450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0842171640 1 -- 172.21.15.107:0/3337465562 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559fb1f70840 msgr2=0x559fb1f34f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0841970640 1 --2- 172.21.15.107:0/3337465562 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559fb1f70840 0x559fb1f34f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0842171640 1 --2- 172.21.15.107:0/3337465562 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559fb1f70840 0x559fb1f34f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0842171640 1 -- 172.21.15.107:0/3337465562 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559fb1d2a490 con 0x559fb1f39960 2023-12-04T05:40:35.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0842171640 1 --2- 172.21.15.107:0/3337465562 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f39960 0x559fb1fd1ce0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0834018040 tx=0x7f0834010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f2c653ebaa6f007 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f081ffff640 1 -- 172.21.15.107:0/3337465562 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083400b6e0 con 0x559fb1f39960 2023-12-04T05:40:35.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f081ffff640 1 -- 172.21.15.107:0/3337465562 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083400b860 con 0x559fb1f39960 2023-12-04T05:40:35.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f081ffff640 1 -- 172.21.15.107:0/3337465562 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083400bc10 con 0x559fb1f39960 2023-12-04T05:40:35.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 1 -- 172.21.15.107:0/3337465562 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f39960 msgr2=0x559fb1fd1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 1 --2- 172.21.15.107:0/3337465562 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f39960 0x559fb1fd1ce0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0834018040 tx=0x7f0834010000 comp rx=0 tx=0).stop 2023-12-04T05:40:35.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 1 -- 172.21.15.107:0/3337465562 shutdown_connections 2023-12-04T05:40:35.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 1 --2- 172.21.15.107:0/3337465562 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f39960 0x559fb1fd1ce0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 1 --2- 172.21.15.107:0/3337465562 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559fb1f39040 0x559fb1f35450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 1 --2- 172.21.15.107:0/3337465562 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559fb1f70840 0x559fb1f34f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 1 -- 172.21.15.107:0/3337465562 >> 172.21.15.107:0/3337465562 conn(0x559fb1f86c20 msgr2=0x559fb1f32640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 1 -- 172.21.15.107:0/3337465562 shutdown_connections 2023-12-04T05:40:35.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 1 -- 172.21.15.107:0/3337465562 wait complete. 2023-12-04T05:40:35.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 1 -- start start 2023-12-04T05:40:35.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 1 Processor -- start 2023-12-04T05:40:35.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559fb1f39040 0x559fb1fd3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f39960 0x559fb1f899b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559fb1f70840 0x559fb1f89ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fb1e0a570 con 0x559fb1f39960 2023-12-04T05:40:35.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fb1dc4440 con 0x559fb1f70840 2023-12-04T05:40:35.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x559fb1d428f0 con 0x559fb1f39040 2023-12-04T05:40:35.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 10 client.0 _ll_register_callbacks cb 0x559fb1d076e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-04T05:40:35.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f084116f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f39960 0x559fb1f899b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f084116f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f39960 0x559fb1f899b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56558/0 (socket says 172.21.15.107:56558) 2023-12-04T05:40:35.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f084116f640 1 -- 172.21.15.107:0/1212368693 learned_addr learned my addr 172.21.15.107:0/1212368693 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0841970640 1 --2- 172.21.15.107:0/1212368693 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559fb1f39040 0x559fb1fd3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f084116f640 1 -- 172.21.15.107:0/1212368693 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559fb1f39040 msgr2=0x559fb1fd3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f084116f640 1 --2- 172.21.15.107:0/1212368693 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559fb1f39040 0x559fb1fd3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f084116f640 1 -- 172.21.15.107:0/1212368693 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559fb1f70840 msgr2=0x559fb1f89ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f084116f640 1 --2- 172.21.15.107:0/1212368693 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559fb1f70840 0x559fb1f89ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f084116f640 1 -- 172.21.15.107:0/1212368693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559fb1a7b890 con 0x559fb1f39960 2023-12-04T05:40:35.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f084116f640 1 --2- 172.21.15.107:0/1212368693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f39960 0x559fb1f899b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f082c09b4d0 tx=0x7f082c01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c36ef13343822ab2 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f081cff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-12-04T05:40:35.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f081cff9640 1 -- 172.21.15.107:0/1212368693 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c0c0070 con 0x559fb1f39960 2023-12-04T05:40:35.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f081cff9640 1 -- 172.21.15.107:0/1212368693 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f082c0bb830 con 0x559fb1f39960 2023-12-04T05:40:35.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 1 -- 172.21.15.107:0/1212368693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559fb1d82110 con 0x559fb1f39960 2023-12-04T05:40:35.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f081cff9640 1 -- 172.21.15.107:0/1212368693 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f082c0b0440 con 0x559fb1f39960 2023-12-04T05:40:35.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 10 client.4698 fetch_fsmap learned FSMap version 17 2023-12-04T05:40:35.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 1 -- 172.21.15.107:0/1212368693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x559fb1a7bcc0 con 0x559fb1f39960 2023-12-04T05:40:35.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f081cff9640 1 -- 172.21.15.107:0/1212368693 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559fb1d82110 con 0x559fb1f39960 2023-12-04T05:40:35.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f081cff9640 1 -- 172.21.15.107:0/1212368693 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 17) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f082c004040 con 0x559fb1f39960 2023-12-04T05:40:35.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 10 client.4698 fetch_fsmap finished waiting for FSMap version 17 2023-12-04T05:40:35.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 10 client.4698 resolve_mds: resolved name 'c' to daemon mds.c 2023-12-04T05:40:35.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 20 client.4698 populate_metadata read hostname 'smithi107' 2023-12-04T05:40:35.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 1 --2- 172.21.15.107:0/1212368693 >> [v2:172.21.15.107:6835/2210527429,v1:172.21.15.107:6837/2210527429] conn(0x559fb1f73fc0 0x559fb1fcd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 4 client.4698 mds_command: new command op to 4263 tid=0 multi_id=0 [{"prefix": "status"}] 2023-12-04T05:40:35.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.550+0000 7f0846d06000 1 -- 172.21.15.107:0/1212368693 --> [v2:172.21.15.107:6835/2210527429,v1:172.21.15.107:6837/2210527429] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559fb1db0650 con 0x559fb1f73fc0 2023-12-04T05:40:35.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0841970640 1 --2- 172.21.15.107:0/1212368693 >> [v2:172.21.15.107:6835/2210527429,v1:172.21.15.107:6837/2210527429] conn(0x559fb1f73fc0 0x559fb1fcd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0841970640 1 --2- 172.21.15.107:0/1212368693 >> [v2:172.21.15.107:6835/2210527429,v1:172.21.15.107:6837/2210527429] conn(0x559fb1f73fc0 0x559fb1fcd250 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=14d0e23e6827b892 server_cookie=d54e2f552ad5da81 in_seq=0 out_seq=0 2023-12-04T05:40:35.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f081cff9640 10 client.4698 ms_handle_connect on v2:172.21.15.107:6835/2210527429 2023-12-04T05:40:35.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f081cff9640 1 -- 172.21.15.107:0/1212368693 <== mds.2 v2:172.21.15.107:6835/2210527429 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x559fb1db0650 con 0x559fb1f73fc0 2023-12-04T05:40:35.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f081cff9640 10 client.4698 handle_command_reply: tid=0 2023-12-04T05:40:35.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0846d06000 1 client.4698 shutdown 2023-12-04T05:40:35.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0846d06000 1 -- 172.21.15.107:0/1212368693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f39960 msgr2=0x559fb1f899b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0846d06000 1 --2- 172.21.15.107:0/1212368693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f39960 0x559fb1f899b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f082c09b4d0 tx=0x7f082c01ba90 comp rx=0 tx=0).stop 2023-12-04T05:40:35.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0846d06000 1 -- 172.21.15.107:0/1212368693 shutdown_connections 2023-12-04T05:40:35.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0846d06000 1 --2- 172.21.15.107:0/1212368693 >> [v2:172.21.15.107:6835/2210527429,v1:172.21.15.107:6837/2210527429] conn(0x559fb1f73fc0 0x559fb1fcd250 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0846d06000 1 --2- 172.21.15.107:0/1212368693 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559fb1f70840 0x559fb1f89ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0846d06000 1 --2- 172.21.15.107:0/1212368693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559fb1f39960 0x559fb1f899b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f081cff9640 0 client.4698 ms_handle_reset on v2:172.21.15.107:6835/2210527429 2023-12-04T05:40:35.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0846d06000 1 --2- 172.21.15.107:0/1212368693 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x559fb1f39040 0x559fb1fd3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0846d06000 1 -- 172.21.15.107:0/1212368693 >> 172.21.15.107:0/1212368693 conn(0x559fb1f86c20 msgr2=0x559fb1fd0690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0846d06000 1 -- 172.21.15.107:0/1212368693 shutdown_connections 2023-12-04T05:40:35.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0846d06000 1 -- 172.21.15.107:0/1212368693 wait complete. 2023-12-04T05:40:35.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0846d06000 20 client.4698 trim_cache size 0 max 16384 2023-12-04T05:40:35.568 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-04T05:40:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "603e856b-31ac-475a-8f35-21ab7faadd48", 2023-12-04T05:40:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 2, 2023-12-04T05:40:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 4263, 2023-12-04T05:40:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-12-04T05:40:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-12-04T05:40:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2023-12-04T05:40:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 23.708028676000001, 2023-12-04T05:40:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 17, 2023-12-04T05:40:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 21, 2023-12-04T05:40:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 21, 2023-12-04T05:40:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 42.316544512999997 2023-12-04T05:40:35.568 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-04T05:40:35.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0843bd4640 1 -- 172.21.15.107:0/3701594430 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f08200668d0 msgr2=0x7f0820068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0843bd4640 1 --2- 172.21.15.107:0/3701594430 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f08200668d0 0x7f0820068d90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f082c0b2000 tx=0x7f082c008040 comp rx=0 tx=0).stop 2023-12-04T05:40:35.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0843bd4640 1 -- 172.21.15.107:0/3701594430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f083c0c8c10 msgr2=0x7f083c154f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0843bd4640 1 --2- 172.21.15.107:0/3701594430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f083c0c8c10 0x7f083c154f00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f083800c820 tx=0x7f083800cce0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0843bd4640 1 -- 172.21.15.107:0/3701594430 shutdown_connections 2023-12-04T05:40:35.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0843bd4640 1 --2- 172.21.15.107:0/3701594430 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f08200668d0 0x7f0820068d90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0843bd4640 1 --2- 172.21.15.107:0/3701594430 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f083c15e6b0 0x7f083c156690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0843bd4640 1 --2- 172.21.15.107:0/3701594430 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f083c1581d0 0x7f083c156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0843bd4640 1 --2- 172.21.15.107:0/3701594430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f083c0c8c10 0x7f083c154f00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0843bd4640 1 -- 172.21.15.107:0/3701594430 >> 172.21.15.107:0/3701594430 conn(0x7f083c0c7ff0 msgr2=0x7f083c150690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0843bd4640 1 -- 172.21.15.107:0/3701594430 shutdown_connections 2023-12-04T05:40:35.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.554+0000 7f0843bd4640 1 -- 172.21.15.107:0/3701594430 wait complete. 2023-12-04T05:40:35.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-12-04T05:40:35.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.686+0000 7f843c174640 1 Processor -- start 2023-12-04T05:40:35.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.686+0000 7f843c174640 1 -- start start 2023-12-04T05:40:35.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.686+0000 7f843c174640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8434151180 0x7f84340c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.686+0000 7f843c174640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8434070380 0x7f8434070790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.686+0000 7f843c174640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84340700c0 con 0x7f84340714f0 2023-12-04T05:40:35.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.686+0000 7f843c174640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8434156df0 con 0x7f8434070380 2023-12-04T05:40:35.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.686+0000 7f843c174640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84341570d0 con 0x7f8434151180 2023-12-04T05:40:35.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.686+0000 7f843a711640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8434070380 0x7f8434070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.686+0000 7f843a711640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8434070380 0x7f8434070790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56570/0 (socket says 172.21.15.107:56570) 2023-12-04T05:40:35.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.686+0000 7f843a711640 1 -- 172.21.15.107:0/3648120966 learned_addr learned my addr 172.21.15.107:0/3648120966 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.686+0000 7f8439f10640 1 --2- 172.21.15.107:0/3648120966 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8434151180 0x7f84340c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.686+0000 7f843a711640 1 -- 172.21.15.107:0/3648120966 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8434151180 msgr2=0x7f84340c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.686+0000 7f843a711640 1 --2- 172.21.15.107:0/3648120966 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8434151180 0x7f84340c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.686+0000 7f843a711640 1 -- 172.21.15.107:0/3648120966 >> v1:172.21.15.195:6789/0 conn(0x7f84340714f0 legacy=0x7f84340718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.686+0000 7f843a711640 1 -- 172.21.15.107:0/3648120966 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84341573b0 con 0x7f8434070380 2023-12-04T05:40:35.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.686+0000 7f8439f10640 1 --2- 172.21.15.107:0/3648120966 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8434151180 0x7f84340c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:35.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f843a711640 1 --2- 172.21.15.107:0/3648120966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8434070380 0x7f8434070790 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f842c009e50 tx=0x7f842c01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fedc03ad417eb036 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f8438f0e640 1 -- 172.21.15.107:0/3648120966 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842c0a5820 con 0x7f8434070380 2023-12-04T05:40:35.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f8438f0e640 1 -- 172.21.15.107:0/3648120966 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842c0a5980 con 0x7f8434070380 2023-12-04T05:40:35.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f8438f0e640 1 -- 172.21.15.107:0/3648120966 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842c0a5d10 con 0x7f8434070380 2023-12-04T05:40:35.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f843c174640 1 -- 172.21.15.107:0/3648120966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8434070380 msgr2=0x7f8434070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f843c174640 1 --2- 172.21.15.107:0/3648120966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8434070380 0x7f8434070790 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f842c009e50 tx=0x7f842c01c9a0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f843c174640 1 -- 172.21.15.107:0/3648120966 shutdown_connections 2023-12-04T05:40:35.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f843c174640 1 --2- 172.21.15.107:0/3648120966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8434070380 0x7f8434070790 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f843c174640 1 --2- 172.21.15.107:0/3648120966 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8434151180 0x7f84340c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f843c174640 1 -- 172.21.15.107:0/3648120966 >> 172.21.15.107:0/3648120966 conn(0x7f84340c7ff0 msgr2=0x7f84340c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f843c174640 1 -- 172.21.15.107:0/3648120966 shutdown_connections 2023-12-04T05:40:35.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f843c174640 1 -- 172.21.15.107:0/3648120966 wait complete. 2023-12-04T05:40:35.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f843c174640 1 Processor -- start 2023-12-04T05:40:35.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f843c174640 1 -- start start 2023-12-04T05:40:35.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f843c174640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8434070380 0x7f843414bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f843c174640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84340714f0 0x7f843413aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f843c174640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8434151180 0x7f843413afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f843c174640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843413d180 con 0x7f8434070380 2023-12-04T05:40:35.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f843c174640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8434156b00 con 0x7f84340714f0 2023-12-04T05:40:35.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f843c174640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f843413d390 con 0x7f8434151180 2023-12-04T05:40:35.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f8439f10640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8434070380 0x7f843414bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f8439f10640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8434070380 0x7f843414bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56578/0 (socket says 172.21.15.107:56578) 2023-12-04T05:40:35.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f8439f10640 1 -- 172.21.15.107:0/879770825 learned_addr learned my addr 172.21.15.107:0/879770825 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f843970f640 1 --2- 172.21.15.107:0/879770825 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84340714f0 0x7f843413aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f843a711640 1 --2- 172.21.15.107:0/879770825 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8434151180 0x7f843413afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f8439f10640 1 -- 172.21.15.107:0/879770825 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8434151180 msgr2=0x7f843413afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f8439f10640 1 --2- 172.21.15.107:0/879770825 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8434151180 0x7f843413afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f8439f10640 1 -- 172.21.15.107:0/879770825 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84340714f0 msgr2=0x7f843413aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f8439f10640 1 --2- 172.21.15.107:0/879770825 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84340714f0 0x7f843413aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f8439f10640 1 -- 172.21.15.107:0/879770825 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f842c009b30 con 0x7f8434070380 2023-12-04T05:40:35.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.690+0000 7f8439f10640 1 --2- 172.21.15.107:0/879770825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8434070380 0x7f843414bf30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f843000c7f0 tx=0x7f843000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.694+0000 7f842a7fc640 1 -- 172.21.15.107:0/879770825 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8430007820 con 0x7f8434070380 2023-12-04T05:40:35.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.694+0000 7f842a7fc640 1 -- 172.21.15.107:0/879770825 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8430007980 con 0x7f8434070380 2023-12-04T05:40:35.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.694+0000 7f843c174640 1 -- 172.21.15.107:0/879770825 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8434156820 con 0x7f8434070380 2023-12-04T05:40:35.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.694+0000 7f842a7fc640 1 -- 172.21.15.107:0/879770825 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8430007d30 con 0x7f8434070380 2023-12-04T05:40:35.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.694+0000 7f843c174640 1 -- 172.21.15.107:0/879770825 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f843413daf0 con 0x7f8434070380 2023-12-04T05:40:35.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.694+0000 7f842a7fc640 1 -- 172.21.15.107:0/879770825 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f84300113f0 con 0x7f8434070380 2023-12-04T05:40:35.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.694+0000 7f842a7fc640 1 --2- 172.21.15.107:0/879770825 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f8418066900 0x7f8418068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.694+0000 7f842a7fc640 1 -- 172.21.15.107:0/879770825 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84300847e0 con 0x7f8434070380 2023-12-04T05:40:35.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.694+0000 7f843970f640 1 --2- 172.21.15.107:0/879770825 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f8418066900 0x7f8418068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.694+0000 7f843970f640 1 --2- 172.21.15.107:0/879770825 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f8418066900 0x7f8418068dc0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f843406dda0 tx=0x7f8424009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.694+0000 7f843f2a6000 1 Processor -- start 2023-12-04T05:40:35.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.694+0000 7f843f2a6000 1 -- start start 2023-12-04T05:40:35.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.694+0000 7f843f2a6000 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55acb60cf040 0x55acb60cf420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.694+0000 7f843f2a6000 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55acb60cf960 0x55acb6109a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.694+0000 7f843f2a6000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb6109fc0 0x55acb610a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.694+0000 7f843f2a6000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acb5f90110 con 0x55acb6109fc0 2023-12-04T05:40:35.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.694+0000 7f843f2a6000 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acb5f209a0 con 0x55acb60cf040 2023-12-04T05:40:35.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.694+0000 7f843f2a6000 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x55acb5f1ded0 con 0x55acb60cf960 2023-12-04T05:40:35.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.694+0000 7f843a711640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb6109fc0 0x55acb610a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.694+0000 7f843a711640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb6109fc0 0x55acb610a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56590/0 (socket says 172.21.15.107:56590) 2023-12-04T05:40:35.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.694+0000 7f843a711640 1 -- 172.21.15.107:0/3516003576 learned_addr learned my addr 172.21.15.107:0/3516003576 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f8439f10640 1 --2- 172.21.15.107:0/3516003576 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55acb60cf040 0x55acb60cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843970f640 1 --2- 172.21.15.107:0/3516003576 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55acb60cf960 0x55acb6109a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843a711640 1 -- 172.21.15.107:0/3516003576 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55acb60cf960 msgr2=0x55acb6109a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843a711640 1 --2- 172.21.15.107:0/3516003576 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55acb60cf960 0x55acb6109a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843a711640 1 -- 172.21.15.107:0/3516003576 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55acb60cf040 msgr2=0x55acb60cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843a711640 1 --2- 172.21.15.107:0/3516003576 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55acb60cf040 0x55acb60cf420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843a711640 1 -- 172.21.15.107:0/3516003576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55acb5c11890 con 0x55acb6109fc0 2023-12-04T05:40:35.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843970f640 1 --2- 172.21.15.107:0/3516003576 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55acb60cf960 0x55acb6109a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:35.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843a711640 1 --2- 172.21.15.107:0/3516003576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb6109fc0 0x55acb610a3a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f842c0af040 tx=0x7f842c004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b88fd295f2586bdd server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f84177fe640 1 -- 172.21.15.107:0/3516003576 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842c0a5cf0 con 0x55acb6109fc0 2023-12-04T05:40:35.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f84177fe640 1 -- 172.21.15.107:0/3516003576 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842c0a5e50 con 0x55acb6109fc0 2023-12-04T05:40:35.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f84177fe640 1 -- 172.21.15.107:0/3516003576 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842c0b9690 con 0x55acb6109fc0 2023-12-04T05:40:35.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843f2a6000 1 -- 172.21.15.107:0/3516003576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb6109fc0 msgr2=0x55acb610a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843f2a6000 1 --2- 172.21.15.107:0/3516003576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb6109fc0 0x55acb610a3a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f842c0af040 tx=0x7f842c004ac0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843f2a6000 1 -- 172.21.15.107:0/3516003576 shutdown_connections 2023-12-04T05:40:35.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843f2a6000 1 --2- 172.21.15.107:0/3516003576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb6109fc0 0x55acb610a3a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843f2a6000 1 --2- 172.21.15.107:0/3516003576 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55acb60cf960 0x55acb6109a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843f2a6000 1 --2- 172.21.15.107:0/3516003576 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55acb60cf040 0x55acb60cf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843f2a6000 1 -- 172.21.15.107:0/3516003576 >> 172.21.15.107:0/3516003576 conn(0x55acb6106840 msgr2=0x55acb6108c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843f2a6000 1 -- 172.21.15.107:0/3516003576 shutdown_connections 2023-12-04T05:40:35.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843f2a6000 1 -- 172.21.15.107:0/3516003576 wait complete. 2023-12-04T05:40:35.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843f2a6000 1 -- start start 2023-12-04T05:40:35.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843f2a6000 1 Processor -- start 2023-12-04T05:40:35.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843f2a6000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb60cf040 0x55acb6104db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843f2a6000 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55acb60cf960 0x55acb61052f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843f2a6000 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55acb6109fc0 0x55acb60cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843f2a6000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acb5f197c0 con 0x55acb60cf040 2023-12-04T05:40:35.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843f2a6000 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acb5ed1bf0 con 0x55acb60cf960 2023-12-04T05:40:35.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843f2a6000 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x55acb5ed7a90 con 0x55acb6109fc0 2023-12-04T05:40:35.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843f2a6000 10 client.0 _ll_register_callbacks cb 0x55acb60b4e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-04T05:40:35.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f8439f10640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb60cf040 0x55acb6104db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f8439f10640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb60cf040 0x55acb6104db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56600/0 (socket says 172.21.15.107:56600) 2023-12-04T05:40:35.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f8439f10640 1 -- 172.21.15.107:0/3324925670 learned_addr learned my addr 172.21.15.107:0/3324925670 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843970f640 1 --2- 172.21.15.107:0/3324925670 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55acb60cf960 0x55acb61052f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f8439f10640 1 -- 172.21.15.107:0/3324925670 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55acb6109fc0 msgr2=0x55acb60cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f8439f10640 1 --2- 172.21.15.107:0/3324925670 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55acb6109fc0 0x55acb60cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f8439f10640 1 -- 172.21.15.107:0/3324925670 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55acb60cf960 msgr2=0x55acb61052f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f8439f10640 1 --2- 172.21.15.107:0/3324925670 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55acb60cf960 0x55acb61052f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f8439f10640 1 -- 172.21.15.107:0/3324925670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84341573b0 con 0x55acb60cf040 2023-12-04T05:40:35.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f8439f10640 1 --2- 172.21.15.107:0/3324925670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb60cf040 0x55acb6104db0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f84300555e0 tx=0x7f8430055e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5c39bda434ef65c server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f8414ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-12-04T05:40:35.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f8414ff9640 1 -- 172.21.15.107:0/3324925670 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8430084040 con 0x55acb60cf040 2023-12-04T05:40:35.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f8414ff9640 1 -- 172.21.15.107:0/3324925670 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f843005d9e0 con 0x55acb60cf040 2023-12-04T05:40:35.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843f2a6000 1 -- 172.21.15.107:0/3324925670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55acb5edb470 con 0x55acb60cf040 2023-12-04T05:40:35.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f8414ff9640 1 -- 172.21.15.107:0/3324925670 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f843005d750 con 0x55acb60cf040 2023-12-04T05:40:35.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843f2a6000 10 client.4710 fetch_fsmap learned FSMap version 17 2023-12-04T05:40:35.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843f2a6000 1 -- 172.21.15.107:0/3324925670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55acb5eda190 con 0x55acb60cf040 2023-12-04T05:40:35.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f8414ff9640 1 -- 172.21.15.107:0/3324925670 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55acb5edb470 con 0x55acb60cf040 2023-12-04T05:40:35.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f8414ff9640 1 -- 172.21.15.107:0/3324925670 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 17) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f8430070070 con 0x55acb60cf040 2023-12-04T05:40:35.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.698+0000 7f843f2a6000 10 client.4710 fetch_fsmap finished waiting for FSMap version 17 2023-12-04T05:40:35.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f843f2a6000 10 client.4710 resolve_mds: resolved name 'd' to daemon mds.d 2023-12-04T05:40:35.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f843f2a6000 20 client.4710 populate_metadata read hostname 'smithi107' 2023-12-04T05:40:35.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f843f2a6000 1 --2- 172.21.15.107:0/3324925670 >> [v2:172.21.15.195:6832/569093002,v1:172.21.15.195:6833/569093002] conn(0x55acb60cc0b0 0x55acb60cc490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f843f2a6000 4 client.4710 mds_command: new command op to 4279 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-12-04T05:40:35.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f843f2a6000 1 -- 172.21.15.107:0/3324925670 --> [v2:172.21.15.195:6832/569093002,v1:172.21.15.195:6833/569093002] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55acb5edeb50 con 0x55acb60cc0b0 2023-12-04T05:40:35.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f843970f640 1 --2- 172.21.15.107:0/3324925670 >> [v2:172.21.15.195:6832/569093002,v1:172.21.15.195:6833/569093002] conn(0x55acb60cc0b0 0x55acb60cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f843970f640 1 --2- 172.21.15.107:0/3324925670 >> [v2:172.21.15.195:6832/569093002,v1:172.21.15.195:6833/569093002] conn(0x55acb60cc0b0 0x55acb60cc490 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f2f111a415b9d0bd server_cookie=4cef77d8478d81f3 in_seq=0 out_seq=0 2023-12-04T05:40:35.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f8414ff9640 10 client.4710 ms_handle_connect on v2:172.21.15.195:6832/569093002 2023-12-04T05:40:35.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f8414ff9640 1 -- 172.21.15.107:0/3324925670 <== mds.1 v2:172.21.15.195:6832/569093002 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55acb5edeb50 con 0x55acb60cc0b0 2023-12-04T05:40:35.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f8414ff9640 10 client.4710 handle_command_reply: tid=0 2023-12-04T05:40:35.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f843f2a6000 1 client.4710 shutdown 2023-12-04T05:40:35.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f843f2a6000 1 -- 172.21.15.107:0/3324925670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb60cf040 msgr2=0x55acb6104db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f843f2a6000 1 --2- 172.21.15.107:0/3324925670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb60cf040 0x55acb6104db0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f84300555e0 tx=0x7f8430055e90 comp rx=0 tx=0).stop 2023-12-04T05:40:35.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f843f2a6000 1 -- 172.21.15.107:0/3324925670 shutdown_connections 2023-12-04T05:40:35.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f843f2a6000 1 --2- 172.21.15.107:0/3324925670 >> [v2:172.21.15.195:6832/569093002,v1:172.21.15.195:6833/569093002] conn(0x55acb60cc0b0 0x55acb60cc490 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f843f2a6000 1 --2- 172.21.15.107:0/3324925670 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55acb6109fc0 0x55acb60cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f843f2a6000 1 --2- 172.21.15.107:0/3324925670 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55acb60cf960 0x55acb61052f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f8414ff9640 0 client.4710 ms_handle_reset on v2:172.21.15.195:6832/569093002 2023-12-04T05:40:35.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f843f2a6000 1 --2- 172.21.15.107:0/3324925670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb60cf040 0x55acb6104db0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f843f2a6000 1 -- 172.21.15.107:0/3324925670 >> 172.21.15.107:0/3324925670 conn(0x55acb6106840 msgr2=0x55acb6109870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f843f2a6000 1 -- 172.21.15.107:0/3324925670 shutdown_connections 2023-12-04T05:40:35.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f843f2a6000 1 -- 172.21.15.107:0/3324925670 wait complete. 2023-12-04T05:40:35.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.702+0000 7f843f2a6000 20 client.4710 trim_cache size 0 max 16384 2023-12-04T05:40:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.718+0000 7f843f2a6000 1 Processor -- start 2023-12-04T05:40:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.718+0000 7f843f2a6000 1 -- start start 2023-12-04T05:40:35.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.718+0000 7f843f2a6000 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55acb60cf040 0x55acb60caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.718+0000 7f843f2a6000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb60cf960 0x55acb60cb480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.718+0000 7f843f2a6000 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55acb6109fc0 0x55acb6172ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.718+0000 7f843f2a6000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acb5bd2940 con 0x55acb60cf960 2023-12-04T05:40:35.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.718+0000 7f843f2a6000 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acb60d1510 con 0x55acb6109fc0 2023-12-04T05:40:35.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.718+0000 7f843f2a6000 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x55acb5ed88f0 con 0x55acb60cf040 2023-12-04T05:40:35.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.718+0000 7f843970f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb60cf960 0x55acb60cb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.718+0000 7f843970f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb60cf960 0x55acb60cb480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56614/0 (socket says 172.21.15.107:56614) 2023-12-04T05:40:35.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.718+0000 7f843970f640 1 -- 172.21.15.107:0/2032091193 learned_addr learned my addr 172.21.15.107:0/2032091193 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.718+0000 7f843970f640 1 -- 172.21.15.107:0/2032091193 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55acb60cf040 msgr2=0x55acb60caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.718+0000 7f843970f640 1 --2- 172.21.15.107:0/2032091193 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55acb60cf040 0x55acb60caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.718+0000 7f843970f640 1 -- 172.21.15.107:0/2032091193 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55acb6109fc0 msgr2=0x55acb6172ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.718+0000 7f843970f640 1 --2- 172.21.15.107:0/2032091193 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55acb6109fc0 0x55acb6172ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.718+0000 7f843970f640 1 -- 172.21.15.107:0/2032091193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55acb5ec0490 con 0x55acb60cf960 2023-12-04T05:40:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.718+0000 7f843970f640 1 --2- 172.21.15.107:0/2032091193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb60cf960 0x55acb60cb480 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f842400e8c0 tx=0x7f8424014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be8ae1cea5b5037a server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f8417fff640 1 -- 172.21.15.107:0/2032091193 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842401a040 con 0x55acb60cf960 2023-12-04T05:40:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f8417fff640 1 -- 172.21.15.107:0/2032091193 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842401b040 con 0x55acb60cf960 2023-12-04T05:40:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f8417fff640 1 -- 172.21.15.107:0/2032091193 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8424015a80 con 0x55acb60cf960 2023-12-04T05:40:35.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 1 -- 172.21.15.107:0/2032091193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb60cf960 msgr2=0x55acb60cb480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 1 --2- 172.21.15.107:0/2032091193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb60cf960 0x55acb60cb480 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f842400e8c0 tx=0x7f8424014000 comp rx=0 tx=0).stop 2023-12-04T05:40:35.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 1 -- 172.21.15.107:0/2032091193 shutdown_connections 2023-12-04T05:40:35.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 1 --2- 172.21.15.107:0/2032091193 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55acb6109fc0 0x55acb6172ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 1 --2- 172.21.15.107:0/2032091193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb60cf960 0x55acb60cb480 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 1 --2- 172.21.15.107:0/2032091193 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55acb60cf040 0x55acb60caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 1 -- 172.21.15.107:0/2032091193 >> 172.21.15.107:0/2032091193 conn(0x55acb6106840 msgr2=0x55acb60c7730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 1 -- 172.21.15.107:0/2032091193 shutdown_connections 2023-12-04T05:40:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 1 -- 172.21.15.107:0/2032091193 wait complete. 2023-12-04T05:40:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 1 -- start start 2023-12-04T05:40:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 1 Processor -- start 2023-12-04T05:40:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55acb60cf040 0x55acb6124ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb60cf960 0x55acb6139740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55acb6109fc0 0x55acb6139c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acb5f15130 con 0x55acb60cf960 2023-12-04T05:40:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acb5fa0570 con 0x55acb60cf040 2023-12-04T05:40:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x55acb5f197c0 con 0x55acb6109fc0 2023-12-04T05:40:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 10 client.0 _ll_register_callbacks cb 0x55acb5e9d6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-04T05:40:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843970f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb60cf960 0x55acb6139740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843970f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb60cf960 0x55acb6139740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56616/0 (socket says 172.21.15.107:56616) 2023-12-04T05:40:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843970f640 1 -- 172.21.15.107:0/226009360 learned_addr learned my addr 172.21.15.107:0/226009360 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843970f640 1 -- 172.21.15.107:0/226009360 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55acb6109fc0 msgr2=0x55acb6139c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843970f640 1 --2- 172.21.15.107:0/226009360 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55acb6109fc0 0x55acb6139c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843970f640 1 -- 172.21.15.107:0/226009360 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55acb60cf040 msgr2=0x55acb6124ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843970f640 1 --2- 172.21.15.107:0/226009360 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55acb60cf040 0x55acb6124ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843970f640 1 -- 172.21.15.107:0/226009360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f842400f040 con 0x55acb60cf960 2023-12-04T05:40:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843970f640 1 --2- 172.21.15.107:0/226009360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb60cf960 0x55acb6139740 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8424028040 tx=0x7f8424003700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4d5bd65e5b75c4a server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f8414ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-12-04T05:40:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f8414ff9640 1 -- 172.21.15.107:0/226009360 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8424015e40 con 0x55acb60cf960 2023-12-04T05:40:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 1 -- 172.21.15.107:0/226009360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55acb5f18110 con 0x55acb60cf960 2023-12-04T05:40:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f8414ff9640 1 -- 172.21.15.107:0/226009360 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8424025e40 con 0x55acb60cf960 2023-12-04T05:40:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f8414ff9640 1 -- 172.21.15.107:0/226009360 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8424003810 con 0x55acb60cf960 2023-12-04T05:40:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 10 client.4716 fetch_fsmap learned FSMap version 17 2023-12-04T05:40:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 1 -- 172.21.15.107:0/226009360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55acb5c11cc0 con 0x55acb60cf960 2023-12-04T05:40:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f8414ff9640 1 -- 172.21.15.107:0/226009360 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55acb5f18110 con 0x55acb60cf960 2023-12-04T05:40:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f8414ff9640 1 -- 172.21.15.107:0/226009360 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 17) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f8424003970 con 0x55acb60cf960 2023-12-04T05:40:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 10 client.4716 fetch_fsmap finished waiting for FSMap version 17 2023-12-04T05:40:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 10 client.4716 resolve_mds: resolved name 'd' to daemon mds.d 2023-12-04T05:40:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 20 client.4716 populate_metadata read hostname 'smithi107' 2023-12-04T05:40:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 1 --2- 172.21.15.107:0/226009360 >> [v2:172.21.15.195:6832/569093002,v1:172.21.15.195:6833/569093002] conn(0x55acb617eb60 0x55acb6180f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 4 client.4716 mds_command: new command op to 4279 tid=0 multi_id=0 [{"prefix": "status"}] 2023-12-04T05:40:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f843f2a6000 1 -- 172.21.15.107:0/226009360 --> [v2:172.21.15.195:6832/569093002,v1:172.21.15.195:6833/569093002] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55acb5f46650 con 0x55acb617eb60 2023-12-04T05:40:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.722+0000 7f8439f10640 1 --2- 172.21.15.107:0/226009360 >> [v2:172.21.15.195:6832/569093002,v1:172.21.15.195:6833/569093002] conn(0x55acb617eb60 0x55acb6180f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f8414ff9640 10 client.4716 ms_handle_connect on v2:172.21.15.195:6832/569093002 2023-12-04T05:40:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f8439f10640 1 --2- 172.21.15.107:0/226009360 >> [v2:172.21.15.195:6832/569093002,v1:172.21.15.195:6833/569093002] conn(0x55acb617eb60 0x55acb6180f50 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=716071a32008fea8 server_cookie=27b0fef30ba2c48 in_seq=0 out_seq=0 2023-12-04T05:40:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f8414ff9640 1 -- 172.21.15.107:0/226009360 <== mds.1 v2:172.21.15.195:6832/569093002 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55acb5f46650 con 0x55acb617eb60 2023-12-04T05:40:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f8414ff9640 10 client.4716 handle_command_reply: tid=0 2023-12-04T05:40:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843f2a6000 1 client.4716 shutdown 2023-12-04T05:40:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843f2a6000 1 -- 172.21.15.107:0/226009360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb60cf960 msgr2=0x55acb6139740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843f2a6000 1 --2- 172.21.15.107:0/226009360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb60cf960 0x55acb6139740 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8424028040 tx=0x7f8424003700 comp rx=0 tx=0).stop 2023-12-04T05:40:35.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843f2a6000 1 -- 172.21.15.107:0/226009360 shutdown_connections 2023-12-04T05:40:35.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843f2a6000 1 --2- 172.21.15.107:0/226009360 >> [v2:172.21.15.195:6832/569093002,v1:172.21.15.195:6833/569093002] conn(0x55acb617eb60 0x55acb6180f50 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843f2a6000 1 --2- 172.21.15.107:0/226009360 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x55acb6109fc0 0x55acb6139c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f8414ff9640 0 client.4716 ms_handle_reset on v2:172.21.15.195:6832/569093002 2023-12-04T05:40:35.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843f2a6000 1 --2- 172.21.15.107:0/226009360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55acb60cf960 0x55acb6139740 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843f2a6000 1 --2- 172.21.15.107:0/226009360 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55acb60cf040 0x55acb6124ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843f2a6000 1 -- 172.21.15.107:0/226009360 >> 172.21.15.107:0/226009360 conn(0x55acb6106840 msgr2=0x55acb60c7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843f2a6000 1 -- 172.21.15.107:0/226009360 shutdown_connections 2023-12-04T05:40:35.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843f2a6000 1 -- 172.21.15.107:0/226009360 wait complete. 2023-12-04T05:40:35.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843f2a6000 20 client.4716 trim_cache size 0 max 16384 2023-12-04T05:40:35.734 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-04T05:40:35.734 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "603e856b-31ac-475a-8f35-21ab7faadd48", 2023-12-04T05:40:35.734 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 1, 2023-12-04T05:40:35.734 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 4279, 2023-12-04T05:40:35.734 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-12-04T05:40:35.734 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-12-04T05:40:35.734 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2023-12-04T05:40:35.734 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 29.924410645999998, 2023-12-04T05:40:35.734 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 17, 2023-12-04T05:40:35.735 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 21, 2023-12-04T05:40:35.735 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 21, 2023-12-04T05:40:35.735 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 42.480492169000001 2023-12-04T05:40:35.735 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-04T05:40:35.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843c174640 1 -- 172.21.15.107:0/879770825 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f8418066900 msgr2=0x7f8418068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843c174640 1 --2- 172.21.15.107:0/879770825 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f8418066900 0x7f8418068dc0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f843406dda0 tx=0x7f8424009040 comp rx=0 tx=0).stop 2023-12-04T05:40:35.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843c174640 1 -- 172.21.15.107:0/879770825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8434070380 msgr2=0x7f843414bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843c174640 1 --2- 172.21.15.107:0/879770825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8434070380 0x7f843414bf30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f843000c7f0 tx=0x7f843000ccb0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843c174640 1 -- 172.21.15.107:0/879770825 shutdown_connections 2023-12-04T05:40:35.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843c174640 1 --2- 172.21.15.107:0/879770825 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f8418066900 0x7f8418068dc0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843c174640 1 --2- 172.21.15.107:0/879770825 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8434151180 0x7f843413afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843c174640 1 --2- 172.21.15.107:0/879770825 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84340714f0 0x7f843413aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843c174640 1 --2- 172.21.15.107:0/879770825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8434070380 0x7f843414bf30 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843c174640 1 -- 172.21.15.107:0/879770825 >> 172.21.15.107:0/879770825 conn(0x7f84340c7ff0 msgr2=0x7f8434072090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843c174640 1 -- 172.21.15.107:0/879770825 shutdown_connections 2023-12-04T05:40:35.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.726+0000 7f843c174640 1 -- 172.21.15.107:0/879770825 wait complete. 2023-12-04T05:40:35.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2023-12-04T05:40:35.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.854+0000 7fb25d6ee640 1 Processor -- start 2023-12-04T05:40:35.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.854+0000 7fb25d6ee640 1 -- start start 2023-12-04T05:40:35.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.854+0000 7fb25d6ee640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb258072110 0x7fb258158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.854+0000 7fb25d6ee640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb2580724f0 0x7fb2580c7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.854+0000 7fb25d6ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb2580c56f0 0x7fb2580c7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.854+0000 7fb25d6ee640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2580c7ea0 con 0x7fb2580c56f0 2023-12-04T05:40:35.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.854+0000 7fb25d6ee640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb258158da0 con 0x7fb2580724f0 2023-12-04T05:40:35.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.854+0000 7fb25d6ee640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2581583d0 con 0x7fb258072110 2023-12-04T05:40:35.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb2577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb2580c56f0 0x7fb2580c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb2577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb2580c56f0 0x7fb2580c7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56622/0 (socket says 172.21.15.107:56622) 2023-12-04T05:40:35.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb2577fe640 1 -- 172.21.15.107:0/2385843560 learned_addr learned my addr 172.21.15.107:0/2385843560 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb256ffd640 1 --2- 172.21.15.107:0/2385843560 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb258072110 0x7fb258158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb2567fc640 1 --2- 172.21.15.107:0/2385843560 >> v2:172.21.15.195:3300/0 conn(0x7fb2580724f0 0x7fb2580c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb2577fe640 1 -- 172.21.15.107:0/2385843560 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb258072110 msgr2=0x7fb258158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb2577fe640 1 --2- 172.21.15.107:0/2385843560 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb258072110 0x7fb258158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb2577fe640 1 -- 172.21.15.107:0/2385843560 >> v2:172.21.15.195:3300/0 conn(0x7fb2580724f0 msgr2=0x7fb2580c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb2577fe640 1 --2- 172.21.15.107:0/2385843560 >> v2:172.21.15.195:3300/0 conn(0x7fb2580724f0 0x7fb2580c7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb2577fe640 1 -- 172.21.15.107:0/2385843560 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2580c6170 con 0x7fb2580c56f0 2023-12-04T05:40:35.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb2567fc640 1 --2- 172.21.15.107:0/2385843560 >> v2:172.21.15.195:3300/0 conn(0x7fb2580724f0 0x7fb2580c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:35.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb2577fe640 1 --2- 172.21.15.107:0/2385843560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2580c56f0 0x7fb2580c7960 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb25806b9c0 tx=0x7fb25001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1a4fed6081c9364 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb255ffb640 1 -- 172.21.15.107:0/2385843560 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2500a5740 con 0x7fb2580c56f0 2023-12-04T05:40:35.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb255ffb640 1 -- 172.21.15.107:0/2385843560 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2500a58a0 con 0x7fb2580c56f0 2023-12-04T05:40:35.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb255ffb640 1 -- 172.21.15.107:0/2385843560 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2500a5c30 con 0x7fb2580c56f0 2023-12-04T05:40:35.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb25d6ee640 1 -- 172.21.15.107:0/2385843560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2580c56f0 msgr2=0x7fb2580c7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb25d6ee640 1 --2- 172.21.15.107:0/2385843560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2580c56f0 0x7fb2580c7960 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb25806b9c0 tx=0x7fb25001c9d0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb25d6ee640 1 -- 172.21.15.107:0/2385843560 shutdown_connections 2023-12-04T05:40:35.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb25d6ee640 1 --2- 172.21.15.107:0/2385843560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2580c56f0 0x7fb2580c7960 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb25d6ee640 1 --2- 172.21.15.107:0/2385843560 >> v2:172.21.15.195:3300/0 conn(0x7fb2580724f0 0x7fb2580c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb25d6ee640 1 --2- 172.21.15.107:0/2385843560 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb258072110 0x7fb258158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb25d6ee640 1 -- 172.21.15.107:0/2385843560 >> 172.21.15.107:0/2385843560 conn(0x7fb2580c7ff0 msgr2=0x7fb2580c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb25d6ee640 1 -- 172.21.15.107:0/2385843560 shutdown_connections 2023-12-04T05:40:35.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.858+0000 7fb25d6ee640 1 -- 172.21.15.107:0/2385843560 wait complete. 2023-12-04T05:40:35.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb25d6ee640 1 Processor -- start 2023-12-04T05:40:35.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb25d6ee640 1 -- start start 2023-12-04T05:40:35.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb25d6ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb258072110 0x7fb258138030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb25d6ee640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb2580724f0 0x7fb25813b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb25d6ee640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2580c56f0 0x7fb2581506b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb25d6ee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2581396f0 con 0x7fb258072110 2023-12-04T05:40:35.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb25d6ee640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2581f3b20 con 0x7fb2580c56f0 2023-12-04T05:40:35.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb25d6ee640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb258139940 con 0x7fb2580724f0 2023-12-04T05:40:35.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb256ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb258072110 0x7fb258138030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb256ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb258072110 0x7fb258138030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56628/0 (socket says 172.21.15.107:56628) 2023-12-04T05:40:35.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb256ffd640 1 -- 172.21.15.107:0/3217736656 learned_addr learned my addr 172.21.15.107:0/3217736656 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb256ffd640 1 -- 172.21.15.107:0/3217736656 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb2580724f0 msgr2=0x7fb25813b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb256ffd640 1 --2- 172.21.15.107:0/3217736656 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb2580724f0 0x7fb25813b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb256ffd640 1 -- 172.21.15.107:0/3217736656 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2580c56f0 msgr2=0x7fb2581506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb256ffd640 1 --2- 172.21.15.107:0/3217736656 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2580c56f0 0x7fb2581506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb256ffd640 1 -- 172.21.15.107:0/3217736656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb250009b30 con 0x7fb258072110 2023-12-04T05:40:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb256ffd640 1 --2- 172.21.15.107:0/3217736656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb258072110 0x7fb258138030 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb24800c7f0 tx=0x7fb24800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb23f7fe640 1 -- 172.21.15.107:0/3217736656 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb248004220 con 0x7fb258072110 2023-12-04T05:40:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb23f7fe640 1 -- 172.21.15.107:0/3217736656 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb248004380 con 0x7fb258072110 2023-12-04T05:40:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb25d6ee640 1 -- 172.21.15.107:0/3217736656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb258139c20 con 0x7fb258072110 2023-12-04T05:40:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb23f7fe640 1 -- 172.21.15.107:0/3217736656 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb248003840 con 0x7fb258072110 2023-12-04T05:40:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb25d6ee640 1 -- 172.21.15.107:0/3217736656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb25813a1e0 con 0x7fb258072110 2023-12-04T05:40:35.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.862+0000 7fb23f7fe640 1 -- 172.21.15.107:0/3217736656 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fb248003bb0 con 0x7fb258072110 2023-12-04T05:40:35.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb23f7fe640 1 --2- 172.21.15.107:0/3217736656 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb2440668d0 0x7fb244068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb23f7fe640 1 -- 172.21.15.107:0/3217736656 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2480843e0 con 0x7fb258072110 2023-12-04T05:40:35.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb2567fc640 1 --2- 172.21.15.107:0/3217736656 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb2440668d0 0x7fb244068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb2567fc640 1 --2- 172.21.15.107:0/3217736656 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb2440668d0 0x7fb244068d90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb25813afd0 tx=0x7fb24c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb260820000 1 Processor -- start 2023-12-04T05:40:35.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb260820000 1 -- start start 2023-12-04T05:40:35.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb260820000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56443696e040 0x56443696e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb260820000 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56443696e960 0x5644369a8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb260820000 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x5644369a8fc0 0x5644369a93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb260820000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x56443682f110 con 0x56443696e040 2023-12-04T05:40:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb260820000 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644367bf9a0 con 0x56443696e960 2023-12-04T05:40:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb260820000 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x5644367bced0 con 0x5644369a8fc0 2023-12-04T05:40:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb256ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56443696e040 0x56443696e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb256ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56443696e040 0x56443696e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56644/0 (socket says 172.21.15.107:56644) 2023-12-04T05:40:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb256ffd640 1 -- 172.21.15.107:0/3139364753 learned_addr learned my addr 172.21.15.107:0/3139364753 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb256ffd640 1 -- 172.21.15.107:0/3139364753 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x5644369a8fc0 msgr2=0x5644369a93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb256ffd640 1 --2- 172.21.15.107:0/3139364753 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x5644369a8fc0 0x5644369a93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb256ffd640 1 -- 172.21.15.107:0/3139364753 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56443696e960 msgr2=0x5644369a8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb256ffd640 1 --2- 172.21.15.107:0/3139364753 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56443696e960 0x5644369a8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb256ffd640 1 -- 172.21.15.107:0/3139364753 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644364b0890 con 0x56443696e040 2023-12-04T05:40:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb256ffd640 1 --2- 172.21.15.107:0/3139364753 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56443696e040 0x56443696e420 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb248082930 tx=0x7fb248054e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61c6855cd9a28ec6 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb23cff9640 1 -- 172.21.15.107:0/3139364753 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24806c070 con 0x56443696e040 2023-12-04T05:40:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb23cff9640 1 -- 172.21.15.107:0/3139364753 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb248055250 con 0x56443696e040 2023-12-04T05:40:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb23cff9640 1 -- 172.21.15.107:0/3139364753 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2480593a0 con 0x56443696e040 2023-12-04T05:40:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb260820000 1 -- 172.21.15.107:0/3139364753 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56443696e040 msgr2=0x56443696e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb260820000 1 --2- 172.21.15.107:0/3139364753 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56443696e040 0x56443696e420 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb248082930 tx=0x7fb248054e90 comp rx=0 tx=0).stop 2023-12-04T05:40:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb260820000 1 -- 172.21.15.107:0/3139364753 shutdown_connections 2023-12-04T05:40:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb260820000 1 --2- 172.21.15.107:0/3139364753 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x5644369a8fc0 0x5644369a93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb260820000 1 --2- 172.21.15.107:0/3139364753 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56443696e960 0x5644369a8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb260820000 1 --2- 172.21.15.107:0/3139364753 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56443696e040 0x56443696e420 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb260820000 1 -- 172.21.15.107:0/3139364753 >> 172.21.15.107:0/3139364753 conn(0x5644369a5840 msgr2=0x5644369a7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb260820000 1 -- 172.21.15.107:0/3139364753 shutdown_connections 2023-12-04T05:40:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb260820000 1 -- 172.21.15.107:0/3139364753 wait complete. 2023-12-04T05:40:35.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb260820000 1 -- start start 2023-12-04T05:40:35.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.866+0000 7fb260820000 1 Processor -- start 2023-12-04T05:40:35.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb260820000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56443696e040 0x5644369a3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb260820000 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x56443696e960 0x5644369a42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb260820000 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5644369a8fc0 0x56443696a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb260820000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644367b87c0 con 0x56443696e040 2023-12-04T05:40:35.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb260820000 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x564436770bf0 con 0x5644369a8fc0 2023-12-04T05:40:35.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb260820000 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x564436776a90 con 0x56443696e960 2023-12-04T05:40:35.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb260820000 10 client.0 _ll_register_callbacks cb 0x564436953e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-04T05:40:35.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb256ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56443696e040 0x5644369a3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb256ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56443696e040 0x5644369a3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56648/0 (socket says 172.21.15.107:56648) 2023-12-04T05:40:35.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb256ffd640 1 -- 172.21.15.107:0/2982980742 learned_addr learned my addr 172.21.15.107:0/2982980742 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb256ffd640 1 -- 172.21.15.107:0/2982980742 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x56443696e960 msgr2=0x5644369a42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb256ffd640 1 --2- 172.21.15.107:0/2982980742 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x56443696e960 0x5644369a42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb256ffd640 1 -- 172.21.15.107:0/2982980742 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5644369a8fc0 msgr2=0x56443696a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb256ffd640 1 --2- 172.21.15.107:0/2982980742 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5644369a8fc0 0x56443696a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb256ffd640 1 -- 172.21.15.107:0/2982980742 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb25813a1e0 con 0x56443696e040 2023-12-04T05:40:35.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb256ffd640 1 --2- 172.21.15.107:0/2982980742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56443696e040 0x5644369a3db0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb2480846c0 tx=0x7fb248000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4dc1d1c4a88747 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb235ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-12-04T05:40:35.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb235ffb640 1 -- 172.21.15.107:0/2982980742 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24806c070 con 0x56443696e040 2023-12-04T05:40:35.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb235ffb640 1 -- 172.21.15.107:0/2982980742 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb24805d5e0 con 0x56443696e040 2023-12-04T05:40:35.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb260820000 1 -- 172.21.15.107:0/2982980742 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56443677a470 con 0x56443696e040 2023-12-04T05:40:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb235ffb640 1 -- 172.21.15.107:0/2982980742 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb248073200 con 0x56443696e040 2023-12-04T05:40:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb260820000 10 client.4728 fetch_fsmap learned FSMap version 17 2023-12-04T05:40:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb260820000 1 -- 172.21.15.107:0/2982980742 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x564436779190 con 0x56443696e040 2023-12-04T05:40:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb235ffb640 1 -- 172.21.15.107:0/2982980742 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56443677a470 con 0x56443696e040 2023-12-04T05:40:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb235ffb640 1 -- 172.21.15.107:0/2982980742 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 17) v1 ==== 4492+0+0 (secure 0 0 0) 0x7fb248051960 con 0x56443696e040 2023-12-04T05:40:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb260820000 10 client.4728 fetch_fsmap finished waiting for FSMap version 17 2023-12-04T05:40:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb260820000 10 client.4728 resolve_mds: resolved name 'f' to daemon mds.f 2023-12-04T05:40:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb260820000 20 client.4728 populate_metadata read hostname 'smithi107' 2023-12-04T05:40:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb260820000 1 --2- 172.21.15.107:0/2982980742 >> [v2:172.21.15.195:6835/3412865477,v1:172.21.15.195:6837/3412865477] conn(0x56443696b140 0x56443696b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb260820000 4 client.4728 mds_command: new command op to 4288 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-12-04T05:40:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb260820000 1 -- 172.21.15.107:0/2982980742 --> [v2:172.21.15.195:6835/3412865477,v1:172.21.15.195:6837/3412865477] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56443677db50 con 0x56443696b140 2023-12-04T05:40:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb2567fc640 1 --2- 172.21.15.107:0/2982980742 >> [v2:172.21.15.195:6835/3412865477,v1:172.21.15.195:6837/3412865477] conn(0x56443696b140 0x56443696b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb235ffb640 10 client.4728 ms_handle_connect on v2:172.21.15.195:6835/3412865477 2023-12-04T05:40:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.870+0000 7fb2567fc640 1 --2- 172.21.15.107:0/2982980742 >> [v2:172.21.15.195:6835/3412865477,v1:172.21.15.195:6837/3412865477] conn(0x56443696b140 0x56443696b520 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56aa6d5ac33a8ccd server_cookie=cb9464fa48947f93 in_seq=0 out_seq=0 2023-12-04T05:40:35.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.874+0000 7fb235ffb640 1 -- 172.21.15.107:0/2982980742 <== mds.0 v2:172.21.15.195:6835/3412865477 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56443677db50 con 0x56443696b140 2023-12-04T05:40:35.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.874+0000 7fb235ffb640 10 client.4728 handle_command_reply: tid=0 2023-12-04T05:40:35.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.874+0000 7fb260820000 1 client.4728 shutdown 2023-12-04T05:40:35.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.874+0000 7fb260820000 1 -- 172.21.15.107:0/2982980742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56443696e040 msgr2=0x5644369a3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.874+0000 7fb260820000 1 --2- 172.21.15.107:0/2982980742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56443696e040 0x5644369a3db0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb2480846c0 tx=0x7fb248000bd0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.874+0000 7fb260820000 1 -- 172.21.15.107:0/2982980742 shutdown_connections 2023-12-04T05:40:35.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.874+0000 7fb260820000 1 --2- 172.21.15.107:0/2982980742 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5644369a8fc0 0x56443696a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.874+0000 7fb260820000 1 --2- 172.21.15.107:0/2982980742 >> [v2:172.21.15.195:6835/3412865477,v1:172.21.15.195:6837/3412865477] conn(0x56443696b140 0x56443696b520 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.874+0000 7fb260820000 1 --2- 172.21.15.107:0/2982980742 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x56443696e960 0x5644369a42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.874+0000 7fb260820000 1 --2- 172.21.15.107:0/2982980742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56443696e040 0x5644369a3db0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.874+0000 7fb235ffb640 0 client.4728 ms_handle_reset on v2:172.21.15.195:6835/3412865477 2023-12-04T05:40:35.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.874+0000 7fb260820000 1 -- 172.21.15.107:0/2982980742 >> 172.21.15.107:0/2982980742 conn(0x5644369a5840 msgr2=0x5644369a8870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.874+0000 7fb260820000 1 -- 172.21.15.107:0/2982980742 shutdown_connections 2023-12-04T05:40:35.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.874+0000 7fb260820000 1 -- 172.21.15.107:0/2982980742 wait complete. 2023-12-04T05:40:35.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.874+0000 7fb260820000 20 client.4728 trim_cache size 0 max 16384 2023-12-04T05:40:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.886+0000 7fb260820000 1 Processor -- start 2023-12-04T05:40:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.886+0000 7fb260820000 1 -- start start 2023-12-04T05:40:35.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.886+0000 7fb260820000 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56443696e040 0x564436a04ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.886+0000 7fb260820000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56443696e960 0x564436a04ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.886+0000 7fb260820000 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x5644369a8fc0 0x564436a05530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.886+0000 7fb260820000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x564436471940 con 0x56443696e960 2023-12-04T05:40:35.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.886+0000 7fb260820000 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644367f9440 con 0x56443696e040 2023-12-04T05:40:35.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.886+0000 7fb260820000 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x5644367b7650 con 0x5644369a8fc0 2023-12-04T05:40:35.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.886+0000 7fb2567fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56443696e960 0x564436a04ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.886+0000 7fb2567fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56443696e960 0x564436a04ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56658/0 (socket says 172.21.15.107:56658) 2023-12-04T05:40:35.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb2567fc640 1 -- 172.21.15.107:0/2944741323 learned_addr learned my addr 172.21.15.107:0/2944741323 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb2567fc640 1 -- 172.21.15.107:0/2944741323 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x5644369a8fc0 msgr2=0x564436a05530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb2567fc640 1 --2- 172.21.15.107:0/2944741323 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x5644369a8fc0 0x564436a05530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb2567fc640 1 -- 172.21.15.107:0/2944741323 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56443696e040 msgr2=0x564436a04ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb2567fc640 1 --2- 172.21.15.107:0/2944741323 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56443696e040 0x564436a04ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb2567fc640 1 -- 172.21.15.107:0/2944741323 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56443675f490 con 0x56443696e960 2023-12-04T05:40:35.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb2567fc640 1 --2- 172.21.15.107:0/2944741323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56443696e960 0x564436a04ff0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb258157a40 tx=0x7fb24c0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=168dd7eb1a7e03b7 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb23d7fa640 1 -- 172.21.15.107:0/2944741323 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24c0152d0 con 0x56443696e960 2023-12-04T05:40:35.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb23d7fa640 1 -- 172.21.15.107:0/2944741323 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb24c0159b0 con 0x56443696e960 2023-12-04T05:40:35.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb23d7fa640 1 -- 172.21.15.107:0/2944741323 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24c015d20 con 0x56443696e960 2023-12-04T05:40:35.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 1 -- 172.21.15.107:0/2944741323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56443696e960 msgr2=0x564436a04ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 1 --2- 172.21.15.107:0/2944741323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56443696e960 0x564436a04ff0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb258157a40 tx=0x7fb24c0046d0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 1 -- 172.21.15.107:0/2944741323 shutdown_connections 2023-12-04T05:40:35.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 1 --2- 172.21.15.107:0/2944741323 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x5644369a8fc0 0x564436a05530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 1 --2- 172.21.15.107:0/2944741323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56443696e960 0x564436a04ff0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 1 --2- 172.21.15.107:0/2944741323 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56443696e040 0x564436a04ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 1 -- 172.21.15.107:0/2944741323 >> 172.21.15.107:0/2944741323 conn(0x5644369a5840 msgr2=0x564436966730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 1 -- 172.21.15.107:0/2944741323 shutdown_connections 2023-12-04T05:40:35.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 1 -- 172.21.15.107:0/2944741323 wait complete. 2023-12-04T05:40:35.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 1 -- start start 2023-12-04T05:40:35.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 1 Processor -- start 2023-12-04T05:40:35.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x56443696e040 0x5644369c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56443696e960 0x5644369c3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644369a8fc0 0x5644369cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x564436771d60 con 0x5644369a8fc0 2023-12-04T05:40:35.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x564436776a90 con 0x56443696e960 2023-12-04T05:40:35.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x56443678db70 con 0x56443696e040 2023-12-04T05:40:35.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 10 client.0 _ll_register_callbacks cb 0x56443673c6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-04T05:40:35.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb2577fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644369a8fc0 0x5644369cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb2577fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644369a8fc0 0x5644369cff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56670/0 (socket says 172.21.15.107:56670) 2023-12-04T05:40:35.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb2577fe640 1 -- 172.21.15.107:0/3367187572 learned_addr learned my addr 172.21.15.107:0/3367187572 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:35.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb256ffd640 1 --2- 172.21.15.107:0/3367187572 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x56443696e040 0x5644369c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb2567fc640 1 --2- 172.21.15.107:0/3367187572 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56443696e960 0x5644369c3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb2577fe640 1 -- 172.21.15.107:0/3367187572 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x56443696e040 msgr2=0x5644369c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb2577fe640 1 --2- 172.21.15.107:0/3367187572 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x56443696e040 0x5644369c36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb2577fe640 1 -- 172.21.15.107:0/3367187572 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56443696e960 msgr2=0x5644369c3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:35.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb2577fe640 1 --2- 172.21.15.107:0/3367187572 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56443696e960 0x5644369c3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb2577fe640 1 -- 172.21.15.107:0/3367187572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5644364b0cc0 con 0x5644369a8fc0 2023-12-04T05:40:35.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb2577fe640 1 --2- 172.21.15.107:0/3367187572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644369a8fc0 0x5644369cff30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb2500af040 tx=0x7fb2500a5a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24f6eae7899506a2 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:35.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb235ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-12-04T05:40:35.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb235ffb640 1 -- 172.21.15.107:0/3367187572 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2500b6460 con 0x5644369a8fc0 2023-12-04T05:40:35.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 1 -- 172.21.15.107:0/3367187572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5644367b7110 con 0x5644369a8fc0 2023-12-04T05:40:35.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb235ffb640 1 -- 172.21.15.107:0/3367187572 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2500b65c0 con 0x5644369a8fc0 2023-12-04T05:40:35.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb235ffb640 1 -- 172.21.15.107:0/3367187572 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2500b67a0 con 0x5644369a8fc0 2023-12-04T05:40:35.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 10 client.4734 fetch_fsmap learned FSMap version 17 2023-12-04T05:40:35.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 1 -- 172.21.15.107:0/3367187572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x5644366e1a50 con 0x5644369a8fc0 2023-12-04T05:40:35.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb235ffb640 1 -- 172.21.15.107:0/3367187572 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5644367b7110 con 0x5644369a8fc0 2023-12-04T05:40:35.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb235ffb640 1 -- 172.21.15.107:0/3367187572 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 17) v1 ==== 4492+0+0 (secure 0 0 0) 0x7fb2500b6900 con 0x5644369a8fc0 2023-12-04T05:40:35.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 10 client.4734 fetch_fsmap finished waiting for FSMap version 17 2023-12-04T05:40:35.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 10 client.4734 resolve_mds: resolved name 'f' to daemon mds.f 2023-12-04T05:40:35.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 20 client.4734 populate_metadata read hostname 'smithi107' 2023-12-04T05:40:35.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 1 --2- 172.21.15.107:0/3367187572 >> [v2:172.21.15.195:6835/3412865477,v1:172.21.15.195:6837/3412865477] conn(0x564436a15350 0x564436a17740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:35.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 4 client.4734 mds_command: new command op to 4288 tid=0 multi_id=0 [{"prefix": "status"}] 2023-12-04T05:40:35.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.890+0000 7fb260820000 1 -- 172.21.15.107:0/3367187572 --> [v2:172.21.15.195:6835/3412865477,v1:172.21.15.195:6837/3412865477] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5644367e5650 con 0x564436a15350 2023-12-04T05:40:35.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb256ffd640 1 --2- 172.21.15.107:0/3367187572 >> [v2:172.21.15.195:6835/3412865477,v1:172.21.15.195:6837/3412865477] conn(0x564436a15350 0x564436a17740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:35.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb235ffb640 10 client.4734 ms_handle_connect on v2:172.21.15.195:6835/3412865477 2023-12-04T05:40:35.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb256ffd640 1 --2- 172.21.15.107:0/3367187572 >> [v2:172.21.15.195:6835/3412865477,v1:172.21.15.195:6837/3412865477] conn(0x564436a15350 0x564436a17740 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b41cceaece49c5 server_cookie=f2d27aca65b0e93f in_seq=0 out_seq=0 2023-12-04T05:40:35.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb235ffb640 1 -- 172.21.15.107:0/3367187572 <== mds.0 v2:172.21.15.195:6835/3412865477 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5644367e5650 con 0x564436a15350 2023-12-04T05:40:35.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb235ffb640 10 client.4734 handle_command_reply: tid=0 2023-12-04T05:40:35.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb260820000 1 client.4734 shutdown 2023-12-04T05:40:35.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb260820000 1 -- 172.21.15.107:0/3367187572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644369a8fc0 msgr2=0x5644369cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb260820000 1 --2- 172.21.15.107:0/3367187572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644369a8fc0 0x5644369cff30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb2500af040 tx=0x7fb2500a5a10 comp rx=0 tx=0).stop 2023-12-04T05:40:35.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb260820000 1 -- 172.21.15.107:0/3367187572 shutdown_connections 2023-12-04T05:40:35.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb260820000 1 --2- 172.21.15.107:0/3367187572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644369a8fc0 0x5644369cff30 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb260820000 1 --2- 172.21.15.107:0/3367187572 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56443696e960 0x5644369c3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb260820000 1 --2- 172.21.15.107:0/3367187572 >> [v2:172.21.15.195:6835/3412865477,v1:172.21.15.195:6837/3412865477] conn(0x564436a15350 0x564436a17740 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb260820000 1 --2- 172.21.15.107:0/3367187572 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x56443696e040 0x5644369c36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb235ffb640 0 client.4734 ms_handle_reset on v2:172.21.15.195:6835/3412865477 2023-12-04T05:40:35.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb260820000 1 -- 172.21.15.107:0/3367187572 >> 172.21.15.107:0/3367187572 conn(0x5644369a5840 msgr2=0x564436966fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb260820000 1 -- 172.21.15.107:0/3367187572 shutdown_connections 2023-12-04T05:40:35.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb260820000 1 -- 172.21.15.107:0/3367187572 wait complete. 2023-12-04T05:40:35.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb260820000 20 client.4734 trim_cache size 0 max 16384 2023-12-04T05:40:35.907 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-04T05:40:35.907 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "603e856b-31ac-475a-8f35-21ab7faadd48", 2023-12-04T05:40:35.907 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2023-12-04T05:40:35.907 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 4288, 2023-12-04T05:40:35.907 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-12-04T05:40:35.907 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-12-04T05:40:35.908 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2023-12-04T05:40:35.908 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 36.144571589000002, 2023-12-04T05:40:35.908 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 17, 2023-12-04T05:40:35.908 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 21, 2023-12-04T05:40:35.908 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 21, 2023-12-04T05:40:35.908 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 42.644830102999997 2023-12-04T05:40:35.908 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-04T05:40:35.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb25d6ee640 1 -- 172.21.15.107:0/3217736656 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb2440668d0 msgr2=0x7fb244068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb25d6ee640 1 --2- 172.21.15.107:0/3217736656 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb2440668d0 0x7fb244068d90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb25813afd0 tx=0x7fb24c008040 comp rx=0 tx=0).stop 2023-12-04T05:40:35.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb25d6ee640 1 -- 172.21.15.107:0/3217736656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb258072110 msgr2=0x7fb258138030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:35.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb25d6ee640 1 --2- 172.21.15.107:0/3217736656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb258072110 0x7fb258138030 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb24800c7f0 tx=0x7fb24800ccb0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb25d6ee640 1 -- 172.21.15.107:0/3217736656 shutdown_connections 2023-12-04T05:40:35.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb25d6ee640 1 --2- 172.21.15.107:0/3217736656 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb2440668d0 0x7fb244068d90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb25d6ee640 1 --2- 172.21.15.107:0/3217736656 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2580c56f0 0x7fb2581506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb25d6ee640 1 --2- 172.21.15.107:0/3217736656 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb2580724f0 0x7fb25813b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb25d6ee640 1 --2- 172.21.15.107:0/3217736656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb258072110 0x7fb258138030 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:35.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb25d6ee640 1 -- 172.21.15.107:0/3217736656 >> 172.21.15.107:0/3217736656 conn(0x7fb2580c7ff0 msgr2=0x7fb2580c6300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:35.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb25d6ee640 1 -- 172.21.15.107:0/3217736656 shutdown_connections 2023-12-04T05:40:35.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:35.894+0000 7fb25d6ee640 1 -- 172.21.15.107:0/3217736656 wait complete. 2023-12-04T05:40:35.913 INFO:teuthology.run_tasks:Running task ceph-fuse... 2023-12-04T05:40:35.938 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2023-12-04T05:40:35.938 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2023-12-04T05:40:35.939 INFO:tasks.ceph_fuse:client.0 config is {} 2023-12-04T05:40:35.939 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-12-04T05:40:35.939 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-04T05:40:35.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:40:35.939 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-12-04T05:40:35.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:40:35.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link delete ceph-brx 2023-12-04T05:40:36.001 INFO:teuthology.orchestra.run.smithi107.stderr:Cannot find device "ceph-brx" 2023-12-04T05:40:36.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T05:40:36.002 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2023-12-04T05:40:36.002 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-04T05:40:36.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.170+0000 7fea50084640 1 Processor -- start 2023-12-04T05:40:36.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.170+0000 7fea50084640 1 -- start start 2023-12-04T05:40:36.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.170+0000 7fea50084640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fea48151180 0x7fea480c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:36.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.170+0000 7fea50084640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fea480714f0 0x7fea480718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:36.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.170+0000 7fea50084640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fea48070380 0x7fea48070790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:36.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.170+0000 7fea50084640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea480700c0 con 0x7fea480714f0 2023-12-04T05:40:36.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.170+0000 7fea50084640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea48156df0 con 0x7fea48070380 2023-12-04T05:40:36.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.170+0000 7fea50084640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea481570d0 con 0x7fea48151180 2023-12-04T05:40:36.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.170+0000 7fea4d61f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fea480714f0 0x7fea480718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:36.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.170+0000 7fea4d61f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fea480714f0 0x7fea480718f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56678/0 (socket says 172.21.15.107:56678) 2023-12-04T05:40:36.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.170+0000 7fea4d61f640 1 -- 172.21.15.107:0/2925645223 learned_addr learned my addr 172.21.15.107:0/2925645223 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:36.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.170+0000 7fea4de20640 1 --2- 172.21.15.107:0/2925645223 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fea48151180 0x7fea480c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:36.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.170+0000 7fea4e621640 1 --2- 172.21.15.107:0/2925645223 >> v2:172.21.15.195:3300/0 conn(0x7fea48070380 0x7fea48070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:36.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.170+0000 7fea4d61f640 1 -- 172.21.15.107:0/2925645223 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fea48151180 msgr2=0x7fea480c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:36.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.170+0000 7fea4d61f640 1 --2- 172.21.15.107:0/2925645223 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fea48151180 0x7fea480c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:36.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.170+0000 7fea4d61f640 1 -- 172.21.15.107:0/2925645223 >> v2:172.21.15.195:3300/0 conn(0x7fea48070380 msgr2=0x7fea48070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:36.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.170+0000 7fea4d61f640 1 --2- 172.21.15.107:0/2925645223 >> v2:172.21.15.195:3300/0 conn(0x7fea48070380 0x7fea48070790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:36.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.170+0000 7fea4d61f640 1 -- 172.21.15.107:0/2925645223 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea481573b0 con 0x7fea480714f0 2023-12-04T05:40:36.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.174+0000 7fea4d61f640 1 --2- 172.21.15.107:0/2925645223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea480714f0 0x7fea480718f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fea4806e5e0 tx=0x7fea3801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7160ca894dc9ac9 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:36.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.174+0000 7fea4ce1e640 1 -- 172.21.15.107:0/2925645223 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea380a89b0 con 0x7fea480714f0 2023-12-04T05:40:36.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.174+0000 7fea4ce1e640 1 -- 172.21.15.107:0/2925645223 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea380a8b10 con 0x7fea480714f0 2023-12-04T05:40:36.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.174+0000 7fea4ce1e640 1 -- 172.21.15.107:0/2925645223 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea380a8ea0 con 0x7fea480714f0 2023-12-04T05:40:36.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.174+0000 7fea50084640 1 -- 172.21.15.107:0/2925645223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea480714f0 msgr2=0x7fea480718f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:36.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.174+0000 7fea50084640 1 --2- 172.21.15.107:0/2925645223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea480714f0 0x7fea480718f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fea4806e5e0 tx=0x7fea3801ecf0 comp rx=0 tx=0).stop 2023-12-04T05:40:36.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.174+0000 7fea50084640 1 -- 172.21.15.107:0/2925645223 shutdown_connections 2023-12-04T05:40:36.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.174+0000 7fea50084640 1 --2- 172.21.15.107:0/2925645223 >> v2:172.21.15.195:3300/0 conn(0x7fea48070380 0x7fea48070790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:36.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.174+0000 7fea50084640 1 --2- 172.21.15.107:0/2925645223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea480714f0 0x7fea480718f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:36.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.174+0000 7fea50084640 1 --2- 172.21.15.107:0/2925645223 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fea48151180 0x7fea480c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:36.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.174+0000 7fea50084640 1 -- 172.21.15.107:0/2925645223 >> 172.21.15.107:0/2925645223 conn(0x7fea480c7ff0 msgr2=0x7fea480c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:36.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.174+0000 7fea50084640 1 -- 172.21.15.107:0/2925645223 shutdown_connections 2023-12-04T05:40:36.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.174+0000 7fea50084640 1 -- 172.21.15.107:0/2925645223 wait complete. 2023-12-04T05:40:36.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea50084640 1 Processor -- start 2023-12-04T05:40:36.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea50084640 1 -- start start 2023-12-04T05:40:36.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea50084640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea48070380 0x7fea48155e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:36.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea50084640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fea480714f0 0x7fea4814a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:36.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea50084640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fea48151180 0x7fea481ed690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:36.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea50084640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea481f0c30 con 0x7fea48070380 2023-12-04T05:40:36.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea50084640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea48146e80 con 0x7fea48151180 2023-12-04T05:40:36.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea50084640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea481f0e80 con 0x7fea480714f0 2023-12-04T05:40:36.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea4de20640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea48070380 0x7fea48155e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:36.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea4de20640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea48070380 0x7fea48155e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56680/0 (socket says 172.21.15.107:56680) 2023-12-04T05:40:36.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea4de20640 1 -- 172.21.15.107:0/86512001 learned_addr learned my addr 172.21.15.107:0/86512001 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:36.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea4de20640 1 -- 172.21.15.107:0/86512001 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fea480714f0 msgr2=0x7fea4814a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:36.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea4de20640 1 --2- 172.21.15.107:0/86512001 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fea480714f0 0x7fea4814a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:36.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea4de20640 1 -- 172.21.15.107:0/86512001 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fea48151180 msgr2=0x7fea481ed690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:36.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea4e621640 1 --2- 172.21.15.107:0/86512001 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fea48151180 0x7fea481ed690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:36.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea4de20640 1 --2- 172.21.15.107:0/86512001 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fea48151180 0x7fea481ed690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:36.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea4de20640 1 -- 172.21.15.107:0/86512001 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea380094d0 con 0x7fea48070380 2023-12-04T05:40:36.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea4de20640 1 --2- 172.21.15.107:0/86512001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea48070380 0x7fea48155e60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fea4400d5f0 tx=0x7fea4400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:36.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea3e7fc640 1 -- 172.21.15.107:0/86512001 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea440125f0 con 0x7fea48070380 2023-12-04T05:40:36.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea3e7fc640 1 -- 172.21.15.107:0/86512001 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea44012750 con 0x7fea48070380 2023-12-04T05:40:36.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea50084640 1 -- 172.21.15.107:0/86512001 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea481f1160 con 0x7fea48070380 2023-12-04T05:40:36.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea3e7fc640 1 -- 172.21.15.107:0/86512001 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea44012ae0 con 0x7fea48070380 2023-12-04T05:40:36.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea50084640 1 -- 172.21.15.107:0/86512001 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea481f1720 con 0x7fea48070380 2023-12-04T05:40:36.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea50084640 1 -- 172.21.15.107:0/86512001 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea48067870 con 0x7fea48070380 2023-12-04T05:40:36.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.178+0000 7fea3e7fc640 1 -- 172.21.15.107:0/86512001 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fea44012c40 con 0x7fea48070380 2023-12-04T05:40:36.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.182+0000 7fea3e7fc640 1 --2- 172.21.15.107:0/86512001 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fea2c0668d0 0x7fea2c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:36.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.182+0000 7fea3e7fc640 1 -- 172.21.15.107:0/86512001 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea440848b0 con 0x7fea48070380 2023-12-04T05:40:36.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.182+0000 7fea4d61f640 1 --2- 172.21.15.107:0/86512001 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fea2c0668d0 0x7fea2c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:36.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.182+0000 7fea4d61f640 1 --2- 172.21.15.107:0/86512001 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fea2c0668d0 0x7fea2c068d90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fea380b2000 tx=0x7fea38004740 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:36.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.186+0000 7fea3e7fc640 1 -- 172.21.15.107:0/86512001 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fea44051b80 con 0x7fea48070380 2023-12-04T05:40:36.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.378+0000 7fea50084640 1 -- 172.21.15.107:0/86512001 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fea48150820 con 0x7fea48070380 2023-12-04T05:40:36.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.378+0000 7fea3e7fc640 1 -- 172.21.15.107:0/86512001 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7fea440551a0 con 0x7fea48070380 2023-12-04T05:40:36.381 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-04T05:40:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.378+0000 7fea50084640 1 -- 172.21.15.107:0/86512001 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fea2c0668d0 msgr2=0x7fea2c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.378+0000 7fea50084640 1 --2- 172.21.15.107:0/86512001 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fea2c0668d0 0x7fea2c068d90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fea380b2000 tx=0x7fea38004740 comp rx=0 tx=0).stop 2023-12-04T05:40:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.378+0000 7fea50084640 1 -- 172.21.15.107:0/86512001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea48070380 msgr2=0x7fea48155e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.378+0000 7fea50084640 1 --2- 172.21.15.107:0/86512001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea48070380 0x7fea48155e60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fea4400d5f0 tx=0x7fea4400dab0 comp rx=0 tx=0).stop 2023-12-04T05:40:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.378+0000 7fea50084640 1 -- 172.21.15.107:0/86512001 shutdown_connections 2023-12-04T05:40:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.378+0000 7fea50084640 1 --2- 172.21.15.107:0/86512001 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fea2c0668d0 0x7fea2c068d90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.378+0000 7fea50084640 1 --2- 172.21.15.107:0/86512001 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fea48151180 0x7fea481ed690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.378+0000 7fea50084640 1 --2- 172.21.15.107:0/86512001 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fea480714f0 0x7fea4814a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.378+0000 7fea50084640 1 --2- 172.21.15.107:0/86512001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea48070380 0x7fea48155e60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.378+0000 7fea50084640 1 -- 172.21.15.107:0/86512001 >> 172.21.15.107:0/86512001 conn(0x7fea480c7ff0 msgr2=0x7fea4815a250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.378+0000 7fea50084640 1 -- 172.21.15.107:0/86512001 shutdown_connections 2023-12-04T05:40:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:36.378+0000 7fea50084640 1 -- 172.21.15.107:0/86512001 wait complete. 2023-12-04T05:40:36.400 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-04T05:40:36.401 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-04T05:40:36.401 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2023-12-04T05:40:36.401 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-04T05:40:36.401 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-04T05:40:36.401 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-04T05:40:36.401 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-04T05:40:36.401 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-04T05:40:36.401 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-04T05:40:36.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:40:36.401 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2023-12-04T05:40:36.407 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-04T05:40:36.407 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-04T05:40:36.407 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2023-12-04T05:40:36.407 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-12-04T05:40:36.407 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2023-12-04T05:40:36.407 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-12-04T05:40:36.407 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-04T05:40:36.407 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2023-12-04T05:40:36.407 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2023-12-04T05:40:36.408 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-04T05:40:36.408 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2023-12-04T05:40:36.408 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2023-12-04T05:40:36.408 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-04T05:40:36.408 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2023-12-04T05:40:36.408 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-04T05:40:36.408 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2023-12-04T05:40:36.408 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-12-04T05:40:36.408 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 41998sec preferred_lft 41998sec 2023-12-04T05:40:36.408 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2023-12-04T05:40:36.408 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-12-04T05:40:36.408 INFO:teuthology.orchestra.run.smithi107.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-04T05:40:36.408 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 02:42:b3:c1:31:38 brd ff:ff:ff:ff:ff:ff 2023-12-04T05:40:36.408 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-04T05:40:36.409 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-12-04T05:40:36.409 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-12-04T05:40:36.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:40:36.409 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-12-04T05:40:36.409 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-12-04T05:40:36.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link add name ceph-brx type bridge 2023-12-04T05:40:36.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip addr flush dev ceph-brx 2023-12-04T05:40:36.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set ceph-brx up 2023-12-04T05:40:36.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-12-04T05:40:36.410 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-12-04T05:40:36.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:40:36.485 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-04T05:40:36.537 INFO:teuthology.orchestra.run.smithi107.stdout:1 2023-12-04T05:40:36.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:40:36.538 DEBUG:teuthology.orchestra.run.smithi107:> route 2023-12-04T05:40:36.608 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2023-12-04T05:40:36.608 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-04T05:40:36.608 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-12-04T05:40:36.608 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-04T05:40:36.608 INFO:teuthology.orchestra.run.smithi107.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-04T05:40:36.608 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-12-04T05:40:36.608 INFO:teuthology.orchestra.run.smithi107.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-04T05:40:36.608 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-04T05:40:36.608 INFO:teuthology.orchestra.run.smithi107.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-04T05:40:36.609 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-04T05:40:36.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:40:36.609 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-12-04T05:40:36.609 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-12-04T05:40:36.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-04T05:40:36.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-04T05:40:36.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-04T05:40:36.609 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-12-04T05:40:36.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:40:36.684 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-12-04T05:40:36.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:40:36.733 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list-id 2023-12-04T05:40:36.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:40:36.780 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-12-04T05:40:36.780 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-12-04T05:40:36.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-04T05:40:36.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-12-04T05:40:36.781 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-12-04T05:40:36.847 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-12-04T05:40:36.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:40:36.848 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-12-04T05:40:36.848 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-12-04T05:40:36.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-12-04T05:40:36.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-12-04T05:40:36.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-12-04T05:40:36.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-12-04T05:40:36.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-12-04T05:40:36.849 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-12-04T05:40:36.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:40:36.968 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-12-04T05:40:36.968 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-12-04T05:40:36.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2023-12-04T05:40:36.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set dev brx.0 master ceph-brx 2023-12-04T05:40:36.968 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-12-04T05:40:36.988 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-04T05:40:36.989 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-04T05:40:36.989 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-04T05:40:37.037 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-04T05:40:37.037 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-04T05:40:37.037 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-04T05:40:37.084 DEBUG:teuthology.orchestra.run.smithi107:> sudo modprobe fuse 2023-12-04T05:40:37.142 DEBUG:teuthology.orchestra.run.smithi107:> cat /proc/self/mounts | awk '{print $2}' 2023-12-04T05:40:37.189 INFO:teuthology.orchestra.run.smithi107.stdout:/sys 2023-12-04T05:40:37.189 INFO:teuthology.orchestra.run.smithi107.stdout:/proc 2023-12-04T05:40:37.189 INFO:teuthology.orchestra.run.smithi107.stdout:/dev 2023-12-04T05:40:37.189 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/pts 2023-12-04T05:40:37.189 INFO:teuthology.orchestra.run.smithi107.stdout:/run 2023-12-04T05:40:37.190 INFO:teuthology.orchestra.run.smithi107.stdout:/ 2023-12-04T05:40:37.190 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/security 2023-12-04T05:40:37.190 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/shm 2023-12-04T05:40:37.190 INFO:teuthology.orchestra.run.smithi107.stdout:/run/lock 2023-12-04T05:40:37.190 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/cgroup 2023-12-04T05:40:37.190 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/pstore 2023-12-04T05:40:37.190 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/bpf 2023-12-04T05:40:37.190 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/sys/fs/binfmt_misc 2023-12-04T05:40:37.190 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/hugepages 2023-12-04T05:40:37.190 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/mqueue 2023-12-04T05:40:37.190 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/debug 2023-12-04T05:40:37.190 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/tracing 2023-12-04T05:40:37.190 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/fs/nfsd 2023-12-04T05:40:37.190 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/fuse/connections 2023-12-04T05:40:37.191 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/config 2023-12-04T05:40:37.191 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-sysusers.service 2023-12-04T05:40:37.191 INFO:teuthology.orchestra.run.smithi107.stdout:/run/rpc_pipefs 2023-12-04T05:40:37.191 INFO:teuthology.orchestra.run.smithi107.stdout:/run/user/1000 2023-12-04T05:40:37.191 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph 2023-12-04T05:40:37.191 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph/osd/ceph-0 2023-12-04T05:40:37.191 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph/osd/ceph-1 2023-12-04T05:40:37.191 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph/osd/ceph-2 2023-12-04T05:40:37.191 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph/osd/ceph-3 2023-12-04T05:40:37.191 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns 2023-12-04T05:40:37.191 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-04T05:40:37.191 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-04T05:40:37.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:40:37.192 DEBUG:teuthology.orchestra.run.smithi107:> ls /sys/fs/fuse/connections 2023-12-04T05:40:37.240 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-04T05:40:37.241 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-04T05:40:37.290 DEBUG:teuthology.orchestra.run.smithi107:> sudo modprobe fuse 2023-12-04T05:40:37.304 DEBUG:teuthology.orchestra.run.smithi107:> cat /proc/self/mounts | awk '{print $2}' 2023-12-04T05:40:37.358 INFO:teuthology.orchestra.run.smithi107.stdout:/sys 2023-12-04T05:40:37.359 INFO:teuthology.orchestra.run.smithi107.stdout:/proc 2023-12-04T05:40:37.359 INFO:teuthology.orchestra.run.smithi107.stdout:/dev 2023-12-04T05:40:37.359 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/pts 2023-12-04T05:40:37.359 INFO:teuthology.orchestra.run.smithi107.stdout:/run 2023-12-04T05:40:37.359 INFO:teuthology.orchestra.run.smithi107.stdout:/ 2023-12-04T05:40:37.359 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/security 2023-12-04T05:40:37.359 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/shm 2023-12-04T05:40:37.359 INFO:teuthology.orchestra.run.smithi107.stdout:/run/lock 2023-12-04T05:40:37.359 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/cgroup 2023-12-04T05:40:37.359 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/pstore 2023-12-04T05:40:37.359 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/bpf 2023-12-04T05:40:37.359 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/sys/fs/binfmt_misc 2023-12-04T05:40:37.359 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/hugepages 2023-12-04T05:40:37.360 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/mqueue 2023-12-04T05:40:37.360 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/debug 2023-12-04T05:40:37.360 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/tracing 2023-12-04T05:40:37.360 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/fs/nfsd 2023-12-04T05:40:37.360 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/fuse/connections 2023-12-04T05:40:37.360 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/config 2023-12-04T05:40:37.360 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-sysusers.service 2023-12-04T05:40:37.360 INFO:teuthology.orchestra.run.smithi107.stdout:/run/rpc_pipefs 2023-12-04T05:40:37.360 INFO:teuthology.orchestra.run.smithi107.stdout:/run/user/1000 2023-12-04T05:40:37.360 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph 2023-12-04T05:40:37.360 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph/osd/ceph-0 2023-12-04T05:40:37.360 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph/osd/ceph-1 2023-12-04T05:40:37.360 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph/osd/ceph-2 2023-12-04T05:40:37.360 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph/osd/ceph-3 2023-12-04T05:40:37.360 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns 2023-12-04T05:40:37.361 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-04T05:40:37.361 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-04T05:40:37.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:40:37.361 DEBUG:teuthology.orchestra.run.smithi107:> ls /sys/fs/fuse/connections 2023-12-04T05:40:37.374 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi107.stderr:2023-12-04T05:40:37.370+0000 7fa626b2b580 -1 init, newargv = 0x55a1d90a6ce0 newargc=15 2023-12-04T05:40:37.374 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi107.stderr:ceph-fuse[46046]: starting ceph client 2023-12-04T05:40:37.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi107.stderr:ceph-fuse[46046]: starting fuse 2023-12-04T05:40:38.480 DEBUG:teuthology.orchestra.run.smithi107:> sudo modprobe fuse 2023-12-04T05:40:38.495 DEBUG:teuthology.orchestra.run.smithi107:> cat /proc/self/mounts | awk '{print $2}' 2023-12-04T05:40:38.542 INFO:teuthology.orchestra.run.smithi107.stdout:/sys 2023-12-04T05:40:38.542 INFO:teuthology.orchestra.run.smithi107.stdout:/proc 2023-12-04T05:40:38.542 INFO:teuthology.orchestra.run.smithi107.stdout:/dev 2023-12-04T05:40:38.542 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/pts 2023-12-04T05:40:38.543 INFO:teuthology.orchestra.run.smithi107.stdout:/run 2023-12-04T05:40:38.543 INFO:teuthology.orchestra.run.smithi107.stdout:/ 2023-12-04T05:40:38.543 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/security 2023-12-04T05:40:38.543 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/shm 2023-12-04T05:40:38.543 INFO:teuthology.orchestra.run.smithi107.stdout:/run/lock 2023-12-04T05:40:38.543 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/cgroup 2023-12-04T05:40:38.543 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/pstore 2023-12-04T05:40:38.543 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/bpf 2023-12-04T05:40:38.543 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/sys/fs/binfmt_misc 2023-12-04T05:40:38.544 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/hugepages 2023-12-04T05:40:38.544 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/mqueue 2023-12-04T05:40:38.544 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/debug 2023-12-04T05:40:38.544 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/tracing 2023-12-04T05:40:38.544 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/fs/nfsd 2023-12-04T05:40:38.544 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/fuse/connections 2023-12-04T05:40:38.544 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/config 2023-12-04T05:40:38.544 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-sysusers.service 2023-12-04T05:40:38.544 INFO:teuthology.orchestra.run.smithi107.stdout:/run/rpc_pipefs 2023-12-04T05:40:38.545 INFO:teuthology.orchestra.run.smithi107.stdout:/run/user/1000 2023-12-04T05:40:38.545 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph 2023-12-04T05:40:38.545 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph/osd/ceph-0 2023-12-04T05:40:38.545 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph/osd/ceph-1 2023-12-04T05:40:38.545 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph/osd/ceph-2 2023-12-04T05:40:38.545 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph/osd/ceph-3 2023-12-04T05:40:38.546 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns 2023-12-04T05:40:38.546 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-04T05:40:38.546 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-04T05:40:38.546 INFO:teuthology.orchestra.run.smithi107.stdout:/home/ubuntu/cephtest/mnt.0 2023-12-04T05:40:38.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:40:38.548 DEBUG:teuthology.orchestra.run.smithi107:> ls /sys/fs/fuse/connections 2023-12-04T05:40:38.597 INFO:teuthology.orchestra.run.smithi107.stdout:45 2023-12-04T05:40:38.598 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2023-12-04T05:40:38.598 DEBUG:teuthology.orchestra.run.smithi107:> sudo stdin-killer -- python3 -c ' 2023-12-04T05:40:38.598 DEBUG:teuthology.orchestra.run.smithi107:> import glob 2023-12-04T05:40:38.598 DEBUG:teuthology.orchestra.run.smithi107:> import re 2023-12-04T05:40:38.598 DEBUG:teuthology.orchestra.run.smithi107:> import os 2023-12-04T05:40:38.598 DEBUG:teuthology.orchestra.run.smithi107:> import subprocess 2023-12-04T05:40:38.598 DEBUG:teuthology.orchestra.run.smithi107:> 2023-12-04T05:40:38.599 DEBUG:teuthology.orchestra.run.smithi107:> def _find_admin_socket(client_name): 2023-12-04T05:40:38.599 DEBUG:teuthology.orchestra.run.smithi107:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-04T05:40:38.599 DEBUG:teuthology.orchestra.run.smithi107:> files = glob.glob(asok_path) 2023-12-04T05:40:38.599 DEBUG:teuthology.orchestra.run.smithi107:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-04T05:40:38.599 DEBUG:teuthology.orchestra.run.smithi107:> 2023-12-04T05:40:38.599 DEBUG:teuthology.orchestra.run.smithi107:> # Given a non-glob path, it better be there 2023-12-04T05:40:38.599 DEBUG:teuthology.orchestra.run.smithi107:> if "*" not in asok_path: 2023-12-04T05:40:38.599 DEBUG:teuthology.orchestra.run.smithi107:> assert(len(files) == 1) 2023-12-04T05:40:38.600 DEBUG:teuthology.orchestra.run.smithi107:> return files[0] 2023-12-04T05:40:38.600 DEBUG:teuthology.orchestra.run.smithi107:> 2023-12-04T05:40:38.600 DEBUG:teuthology.orchestra.run.smithi107:> for f in files: 2023-12-04T05:40:38.600 DEBUG:teuthology.orchestra.run.smithi107:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-04T05:40:38.600 DEBUG:teuthology.orchestra.run.smithi107:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-04T05:40:38.600 DEBUG:teuthology.orchestra.run.smithi107:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-04T05:40:38.600 DEBUG:teuthology.orchestra.run.smithi107:> contents = proc_f.read() 2023-12-04T05:40:38.601 DEBUG:teuthology.orchestra.run.smithi107:> if mountpoint in contents: 2023-12-04T05:40:38.601 DEBUG:teuthology.orchestra.run.smithi107:> return f 2023-12-04T05:40:38.601 DEBUG:teuthology.orchestra.run.smithi107:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-04T05:40:38.601 DEBUG:teuthology.orchestra.run.smithi107:> 2023-12-04T05:40:38.601 DEBUG:teuthology.orchestra.run.smithi107:> print(_find_admin_socket("client.0")) 2023-12-04T05:40:38.601 DEBUG:teuthology.orchestra.run.smithi107:> ' 2023-12-04T05:40:38.722 INFO:teuthology.orchestra.run.smithi107.stdout:/var/run/ceph/ceph-client.0.46046.asok 2023-12-04T05:40:38.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-04T05:40:38.734 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.46046.asok 2023-12-04T05:40:38.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:40:38.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.46046.asok status 2023-12-04T05:40:38.884 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-04T05:40:38.884 INFO:teuthology.orchestra.run.smithi107.stdout: "metadata": { 2023-12-04T05:40:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_sha1": "47c1c547372ed5bff7984647dc200bac0e019caf", 2023-12-04T05:40:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_version": "ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev)", 2023-12-04T05:40:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: "entity_id": "0", 2023-12-04T05:40:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: "hostname": "smithi107", 2023-12-04T05:40:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-04T05:40:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: "pid": "46046", 2023-12-04T05:40:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: "root": "/" 2023-12-04T05:40:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-04T05:40:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: "dentry_count": 0, 2023-12-04T05:40:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: "dentry_pinned_count": 0, 2023-12-04T05:40:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 4749, 2023-12-04T05:40:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: "inst": { 2023-12-04T05:40:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: "name": { 2023-12-04T05:40:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client", 2023-12-04T05:40:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: "num": 4749 2023-12-04T05:40:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-04T05:40:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": { 2023-12-04T05:40:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2023-12-04T05:40:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "192.168.0.1:0", 2023-12-04T05:40:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1451854371 2023-12-04T05:40:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-04T05:40:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-04T05:40:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": { 2023-12-04T05:40:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2023-12-04T05:40:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "192.168.0.1:0", 2023-12-04T05:40:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1451854371 2023-12-04T05:40:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-04T05:40:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: "inst_str": "client.4749 192.168.0.1:0/1451854371", 2023-12-04T05:40:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: "addr_str": "192.168.0.1:0/1451854371", 2023-12-04T05:40:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: "inode_count": 1, 2023-12-04T05:40:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_epoch": 17, 2023-12-04T05:40:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_epoch": 21, 2023-12-04T05:40:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_epoch_barrier": 0, 2023-12-04T05:40:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: "blocklisted": false, 2023-12-04T05:40:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs" 2023-12-04T05:40:38.887 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-04T05:40:38.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:40:38.899 DEBUG:teuthology.orchestra.run.smithi107:> stat --file-system '--printf=%T 2023-12-04T05:40:38.899 DEBUG:teuthology.orchestra.run.smithi107:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-04T05:40:38.904 INFO:teuthology.orchestra.run.smithi107.stdout:fuseblk 2023-12-04T05:40:38.904 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-04T05:40:38.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:40:38.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-04T05:40:38.963 INFO:teuthology.run_tasks:Running task mds_thrash... 2023-12-04T05:40:38.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T05:40:39.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.118+0000 7fed24549640 1 Processor -- start 2023-12-04T05:40:39.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.118+0000 7fed24549640 1 -- start start 2023-12-04T05:40:39.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.122+0000 7fed24549640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fed1c072110 0x7fed1c158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:39.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.122+0000 7fed24549640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fed1c0724f0 0x7fed1c0c7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:39.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.122+0000 7fed24549640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed1c158da0 con 0x7fed1c0c56f0 2023-12-04T05:40:39.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.122+0000 7fed24549640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed1c1583d0 con 0x7fed1c072110 2023-12-04T05:40:39.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.122+0000 7fed24549640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c0c6170 con 0x7fed1c0724f0 2023-12-04T05:40:39.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.122+0000 7fed222e5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fed1c072110 0x7fed1c158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:39.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.122+0000 7fed22ae6640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fed1c0c56f0 0x7fed1c140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39584/0 (socket says 172.21.15.107:39584) 2023-12-04T05:40:39.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.122+0000 7fed22ae6640 1 -- 172.21.15.107:0/4079583520 learned_addr learned my addr 172.21.15.107:0/4079583520 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:39.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.122+0000 7fed21ae4640 1 --2- 172.21.15.107:0/4079583520 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fed1c0724f0 0x7fed1c0c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:39.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.122+0000 7fed222e5640 1 -- 172.21.15.107:0/4079583520 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fed1c0724f0 msgr2=0x7fed1c0c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:39.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.122+0000 7fed222e5640 1 --2- 172.21.15.107:0/4079583520 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fed1c0724f0 0x7fed1c0c7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.122+0000 7fed222e5640 1 -- 172.21.15.107:0/4079583520 >> v1:172.21.15.107:6789/0 conn(0x7fed1c0c56f0 legacy=0x7fed1c140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:39.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.122+0000 7fed222e5640 1 -- 172.21.15.107:0/4079583520 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1c157da0 con 0x7fed1c072110 2023-12-04T05:40:39.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.122+0000 7fed212e3640 1 -- 172.21.15.107:0/4079583520 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086379054 0 0) 0x7fed140014a0 con 0x7fed1c0c56f0 2023-12-04T05:40:39.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.122+0000 7fed21ae4640 1 --2- 172.21.15.107:0/4079583520 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fed1c0724f0 0x7fed1c0c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:40:39.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.122+0000 7fed222e5640 1 --2- 172.21.15.107:0/4079583520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed1c072110 0x7fed1c158860 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fed100097c0 tx=0x7fed1001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec0002a23983a2b server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:39.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed212e3640 1 -- 172.21.15.107:0/4079583520 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed100a8970 con 0x7fed1c072110 2023-12-04T05:40:39.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed212e3640 1 -- 172.21.15.107:0/4079583520 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed100a8ad0 con 0x7fed1c072110 2023-12-04T05:40:39.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed212e3640 1 -- 172.21.15.107:0/4079583520 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed100a8e60 con 0x7fed1c072110 2023-12-04T05:40:39.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed24549640 1 -- 172.21.15.107:0/4079583520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed1c072110 msgr2=0x7fed1c158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:39.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed24549640 1 --2- 172.21.15.107:0/4079583520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed1c072110 0x7fed1c158860 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fed100097c0 tx=0x7fed1001ec70 comp rx=0 tx=0).stop 2023-12-04T05:40:39.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed24549640 1 -- 172.21.15.107:0/4079583520 shutdown_connections 2023-12-04T05:40:39.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed24549640 1 --2- 172.21.15.107:0/4079583520 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fed1c0724f0 0x7fed1c0c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed24549640 1 --2- 172.21.15.107:0/4079583520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed1c072110 0x7fed1c158860 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed24549640 1 -- 172.21.15.107:0/4079583520 >> 172.21.15.107:0/4079583520 conn(0x7fed1c0c7ff0 msgr2=0x7fed1c0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:39.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed24549640 1 -- 172.21.15.107:0/4079583520 shutdown_connections 2023-12-04T05:40:39.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed24549640 1 -- 172.21.15.107:0/4079583520 wait complete. 2023-12-04T05:40:39.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed24549640 1 Processor -- start 2023-12-04T05:40:39.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed24549640 1 -- start start 2023-12-04T05:40:39.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed24549640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fed1c072110 0x7fed1c1edb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:39.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed24549640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fed1c0724f0 0x7fed1c157020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:39.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed24549640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed1c0c56f0 0x7fed1c14ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:39.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed24549640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c155120 con 0x7fed1c0c56f0 2023-12-04T05:40:39.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed24549640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c15bb40 con 0x7fed1c072110 2023-12-04T05:40:39.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed24549640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c1ee080 con 0x7fed1c0724f0 2023-12-04T05:40:39.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed22ae6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed1c0c56f0 0x7fed1c14ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:39.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed22ae6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed1c0c56f0 0x7fed1c14ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56160/0 (socket says 172.21.15.107:56160) 2023-12-04T05:40:39.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed22ae6640 1 -- 172.21.15.107:0/4020537218 learned_addr learned my addr 172.21.15.107:0/4020537218 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:39.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed222e5640 1 --2- 172.21.15.107:0/4020537218 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fed1c072110 0x7fed1c1edb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:39.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed22ae6640 1 -- 172.21.15.107:0/4020537218 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fed1c0724f0 msgr2=0x7fed1c157020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:39.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed21ae4640 1 --2- 172.21.15.107:0/4020537218 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fed1c0724f0 0x7fed1c157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:39.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed22ae6640 1 --2- 172.21.15.107:0/4020537218 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fed1c0724f0 0x7fed1c157020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed22ae6640 1 -- 172.21.15.107:0/4020537218 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fed1c072110 msgr2=0x7fed1c1edb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:39.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed22ae6640 1 --2- 172.21.15.107:0/4020537218 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fed1c072110 0x7fed1c1edb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed22ae6640 1 -- 172.21.15.107:0/4020537218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed10009420 con 0x7fed1c0c56f0 2023-12-04T05:40:39.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed222e5640 1 --2- 172.21.15.107:0/4020537218 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fed1c072110 0x7fed1c1edb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:39.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed22ae6640 1 --2- 172.21.15.107:0/4020537218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed1c0c56f0 0x7fed1c14ce40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fed1400c840 tx=0x7fed1400cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:39.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed0effd640 1 -- 172.21.15.107:0/4020537218 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1400a640 con 0x7fed1c0c56f0 2023-12-04T05:40:39.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed0effd640 1 -- 172.21.15.107:0/4020537218 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1400a7c0 con 0x7fed1c0c56f0 2023-12-04T05:40:39.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed0effd640 1 -- 172.21.15.107:0/4020537218 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1400ab70 con 0x7fed1c0c56f0 2023-12-04T05:40:39.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed24549640 1 -- 172.21.15.107:0/4020537218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed1c154300 con 0x7fed1c0c56f0 2023-12-04T05:40:39.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.126+0000 7fed24549640 1 -- 172.21.15.107:0/4020537218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1c154880 con 0x7fed1c0c56f0 2023-12-04T05:40:39.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.130+0000 7fed24549640 1 -- 172.21.15.107:0/4020537218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed1c067870 con 0x7fed1c0c56f0 2023-12-04T05:40:39.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.130+0000 7fed0effd640 1 -- 172.21.15.107:0/4020537218 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fed1400acd0 con 0x7fed1c0c56f0 2023-12-04T05:40:39.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.130+0000 7fed0effd640 1 --2- 172.21.15.107:0/4020537218 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fed000668d0 0x7fed00068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:39.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.130+0000 7fed222e5640 1 --2- 172.21.15.107:0/4020537218 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fed000668d0 0x7fed00068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:39.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.130+0000 7fed0effd640 1 -- 172.21.15.107:0/4020537218 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed14083820 con 0x7fed1c0c56f0 2023-12-04T05:40:39.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.130+0000 7fed222e5640 1 --2- 172.21.15.107:0/4020537218 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fed000668d0 0x7fed00068d90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fed100b2000 tx=0x7fed100aa070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:39.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.134+0000 7fed0effd640 1 -- 172.21.15.107:0/4020537218 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fed140509d0 con 0x7fed1c0c56f0 2023-12-04T05:40:39.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.290+0000 7fed24549640 1 -- 172.21.15.107:0/4020537218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed1c158da0 con 0x7fed1c0c56f0 2023-12-04T05:40:39.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.290+0000 7fed0effd640 1 -- 172.21.15.107:0/4020537218 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7fed1c158da0 con 0x7fed1c0c56f0 2023-12-04T05:40:39.296 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:39.296 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":21,"fsid":"603e856b-31ac-475a-8f35-21ab7faadd48","created":"2023-12-04T05:39:34.921690+0000","modified":"2023-12-04T05:40:00.671650+0000","last_up_change":"2023-12-04T05:39:46.235790+0000","last_in_change":"2023-12-04T05:39:37.587639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T05:39:47.567702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T05:39:57.644286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T05:39:58.029897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ab7111c-0ae8-4d41-b6db-e3743614322c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6817","nonce":768714279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6821","nonce":768714279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6829","nonce":768714279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6825","nonce":768714279}]},"public_addr":"172.21.15.107:6817/768714279","cluster_addr":"172.21.15.107:6821/768714279","heartbeat_back_addr":"172.21.15.107:6829/768714279","heartbeat_front_addr":"172.21.15.107:6825/768714279","state":["exists","up"]},{"osd":1,"uuid":"baef7a1f-8026-4fab-9e70-48465271034e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6801","nonce":2496355981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6803","nonce":2496355981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6807","nonce":2496355981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6805","nonce":2496355981}]},"public_addr":"172.21.15.107:6801/2496355981","cluster_addr":"172.21.15.107:6803/2496355981","heartbeat_back_addr":"172.21.15.107:6807/2496355981","heartbeat_front_addr":"172.21.15.107:6805/2496355981","state":["exists","up"]},{"osd":2,"uuid":"90e8cf24-7b54-484a-9d85-b44a9d3dfeab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6809","nonce":3097298224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6811","nonce":3097298224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6815","nonce":3097298224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6813","nonce":3097298224}]},"public_addr":"172.21.15.107:6809/3097298224","cluster_addr":"172.21.15.107:6811/3097298224","heartbeat_back_addr":"172.21.15.107:6815/3097298224","heartbeat_front_addr":"172.21.15.107:6813/3097298224","state":["exists","up"]},{"osd":3,"uuid":"54658731-2b9a-4d5e-ab13-7c1be7d34e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6820","nonce":3773076671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6824","nonce":3773076671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6831","nonce":3773076671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6828","nonce":3773076671}]},"public_addr":"172.21.15.107:6820/3773076671","cluster_addr":"172.21.15.107:6824/3773076671","heartbeat_back_addr":"172.21.15.107:6831/3773076671","heartbeat_front_addr":"172.21.15.107:6828/3773076671","state":["exists","up"]},{"osd":4,"uuid":"9a9dea13-8afe-485d-aa43-45eda691179b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6801","nonce":2747823542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6803","nonce":2747823542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6807","nonce":2747823542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6805","nonce":2747823542}]},"public_addr":"172.21.15.195:6801/2747823542","cluster_addr":"172.21.15.195:6803/2747823542","heartbeat_back_addr":"172.21.15.195:6807/2747823542","heartbeat_front_addr":"172.21.15.195:6805/2747823542","state":["exists","up"]},{"osd":5,"uuid":"db840d83-5a7d-468a-8aa2-1997945847c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6825","nonce":1484264540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6827","nonce":1484264540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6831","nonce":1484264540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6829","nonce":1484264540}]},"public_addr":"172.21.15.195:6825/1484264540","cluster_addr":"172.21.15.195:6827/1484264540","heartbeat_back_addr":"172.21.15.195:6831/1484264540","heartbeat_front_addr":"172.21.15.195:6829/1484264540","state":["exists","up"]},{"osd":6,"uuid":"38b5f165-bf91-467e-a615-4b4784337689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6811","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6813","nonce":2638471412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6817","nonce":2638471412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6823","nonce":2638471412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6821","nonce":2638471412}]},"public_addr":"172.21.15.195:6813/2638471412","cluster_addr":"172.21.15.195:6817/2638471412","heartbeat_back_addr":"172.21.15.195:6823/2638471412","heartbeat_front_addr":"172.21.15.195:6821/2638471412","state":["exists","up"]},{"osd":7,"uuid":"e5b3c596-494b-4df3-9437-bc2d52168ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6809","nonce":768189614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6812","nonce":768189614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6820","nonce":768189614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6816","nonce":768189614}]},"public_addr":"172.21.15.195:6809/768189614","cluster_addr":"172.21.15.195:6812/768189614","heartbeat_back_addr":"172.21.15.195:6820/768189614","heartbeat_front_addr":"172.21.15.195:6816/768189614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.209077+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.086601+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.221568+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.238524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.059477+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.148422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.079867+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.064916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T05:40:39.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.294+0000 7fed24549640 1 -- 172.21.15.107:0/4020537218 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fed000668d0 msgr2=0x7fed00068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:39.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.294+0000 7fed24549640 1 --2- 172.21.15.107:0/4020537218 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fed000668d0 0x7fed00068d90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fed100b2000 tx=0x7fed100aa070 comp rx=0 tx=0).stop 2023-12-04T05:40:39.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.294+0000 7fed24549640 1 -- 172.21.15.107:0/4020537218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed1c0c56f0 msgr2=0x7fed1c14ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:39.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.294+0000 7fed24549640 1 --2- 172.21.15.107:0/4020537218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed1c0c56f0 0x7fed1c14ce40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fed1400c840 tx=0x7fed1400cd00 comp rx=0 tx=0).stop 2023-12-04T05:40:39.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.294+0000 7fed24549640 1 -- 172.21.15.107:0/4020537218 shutdown_connections 2023-12-04T05:40:39.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.294+0000 7fed24549640 1 --2- 172.21.15.107:0/4020537218 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fed000668d0 0x7fed00068d90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.294+0000 7fed24549640 1 --2- 172.21.15.107:0/4020537218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed1c0c56f0 0x7fed1c14ce40 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.294+0000 7fed24549640 1 --2- 172.21.15.107:0/4020537218 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fed1c0724f0 0x7fed1c157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.294+0000 7fed24549640 1 --2- 172.21.15.107:0/4020537218 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fed1c072110 0x7fed1c1edb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.294+0000 7fed24549640 1 -- 172.21.15.107:0/4020537218 >> 172.21.15.107:0/4020537218 conn(0x7fed1c0c7ff0 msgr2=0x7fed1c0c2160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:39.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.294+0000 7fed24549640 1 -- 172.21.15.107:0/4020537218 shutdown_connections 2023-12-04T05:40:39.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.294+0000 7fed24549640 1 -- 172.21.15.107:0/4020537218 wait complete. 2023-12-04T05:40:39.314 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-04T05:39:47.567702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-04T05:39:57.644286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-04T05:39:58.029897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-04T05:40:39.315 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-04T05:40:39.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.422+0000 7f456f4d3640 1 Processor -- start 2023-12-04T05:40:39.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.422+0000 7f456f4d3640 1 -- start start 2023-12-04T05:40:39.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.426+0000 7f456f4d3640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4568151180 0x7f45680c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:39.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.426+0000 7f456f4d3640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45680714f0 0x7f45680718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:39.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.426+0000 7f456f4d3640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45680700c0 con 0x7f4568070380 2023-12-04T05:40:39.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.426+0000 7f456f4d3640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4568156df0 con 0x7f4568151180 2023-12-04T05:40:39.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.426+0000 7f456f4d3640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45681570d0 con 0x7f45680714f0 2023-12-04T05:40:39.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.426+0000 7f456ca6e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45680714f0 0x7f45680718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:39.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.426+0000 7f456d26f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4568151180 0x7f45680c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:39.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.426+0000 7f456ca6e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45680714f0 0x7f45680718f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.107:60588/0 (socket says 172.21.15.107:60588) 2023-12-04T05:40:39.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.426+0000 7f456ca6e640 1 -- 172.21.15.107:0/965420839 learned_addr learned my addr 172.21.15.107:0/965420839 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:39.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.426+0000 7f456da70640 1 --1- 172.21.15.107:0/965420839 >> v1:172.21.15.195:6789/0 conn(0x7f4568070380 0x7f4568070790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:50582/0 (socket says 172.21.15.107:50582) 2023-12-04T05:40:39.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.426+0000 7f456ca6e640 1 -- 172.21.15.107:0/965420839 >> v2:172.21.15.195:3300/0 conn(0x7f4568151180 msgr2=0x7f45680c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:39.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.426+0000 7f456ca6e640 1 --2- 172.21.15.107:0/965420839 >> v2:172.21.15.195:3300/0 conn(0x7f4568151180 0x7f45680c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.426+0000 7f456ca6e640 1 -- 172.21.15.107:0/965420839 >> v1:172.21.15.195:6789/0 conn(0x7f4568070380 legacy=0x7f4568070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:39.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.426+0000 7f456ca6e640 1 -- 172.21.15.107:0/965420839 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45681573b0 con 0x7f45680714f0 2023-12-04T05:40:39.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.426+0000 7f456d26f640 1 --2- 172.21.15.107:0/965420839 >> v2:172.21.15.195:3300/0 conn(0x7f4568151180 0x7f45680c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T05:40:39.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.426+0000 7f456ca6e640 1 --2- 172.21.15.107:0/965420839 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45680714f0 0x7f45680718f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4558009870 tx=0x7f455801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef0f1a94d4fbb56c server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:39.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f455ffff640 1 -- 172.21.15.107:0/965420839 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45580a8a30 con 0x7f45680714f0 2023-12-04T05:40:39.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f455ffff640 1 -- 172.21.15.107:0/965420839 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45580a8b90 con 0x7f45680714f0 2023-12-04T05:40:39.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f455ffff640 1 -- 172.21.15.107:0/965420839 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45580b0440 con 0x7f45680714f0 2023-12-04T05:40:39.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456f4d3640 1 -- 172.21.15.107:0/965420839 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45680714f0 msgr2=0x7f45680718f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:39.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456f4d3640 1 --2- 172.21.15.107:0/965420839 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45680714f0 0x7f45680718f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4558009870 tx=0x7f455801ed30 comp rx=0 tx=0).stop 2023-12-04T05:40:39.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456f4d3640 1 -- 172.21.15.107:0/965420839 shutdown_connections 2023-12-04T05:40:39.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456f4d3640 1 --2- 172.21.15.107:0/965420839 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f45680714f0 0x7f45680718f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456f4d3640 1 --2- 172.21.15.107:0/965420839 >> v2:172.21.15.195:3300/0 conn(0x7f4568151180 0x7f45680c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456f4d3640 1 -- 172.21.15.107:0/965420839 >> 172.21.15.107:0/965420839 conn(0x7f45680c7ff0 msgr2=0x7f45680c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:39.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456f4d3640 1 -- 172.21.15.107:0/965420839 shutdown_connections 2023-12-04T05:40:39.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456f4d3640 1 -- 172.21.15.107:0/965420839 wait complete. 2023-12-04T05:40:39.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456f4d3640 1 Processor -- start 2023-12-04T05:40:39.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456f4d3640 1 -- start start 2023-12-04T05:40:39.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456f4d3640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4568070380 0x7f4568159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:39.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456f4d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45680714f0 0x7f4568159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:39.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456f4d3640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4568151180 0x7f456815a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:39.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456f4d3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4568158b70 con 0x7f45680714f0 2023-12-04T05:40:39.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456f4d3640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4568159a80 con 0x7f4568070380 2023-12-04T05:40:39.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456f4d3640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45681ed870 con 0x7f4568151180 2023-12-04T05:40:39.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456ca6e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45680714f0 0x7f4568159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:39.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456ca6e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45680714f0 0x7f4568159540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56164/0 (socket says 172.21.15.107:56164) 2023-12-04T05:40:39.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456ca6e640 1 -- 172.21.15.107:0/3806016347 learned_addr learned my addr 172.21.15.107:0/3806016347 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:39.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456d26f640 1 --2- 172.21.15.107:0/3806016347 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4568070380 0x7f4568159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:39.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456da70640 1 --2- 172.21.15.107:0/3806016347 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4568151180 0x7f456815a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:39.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456ca6e640 1 -- 172.21.15.107:0/3806016347 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4568151180 msgr2=0x7f456815a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:39.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456ca6e640 1 --2- 172.21.15.107:0/3806016347 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4568151180 0x7f456815a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456ca6e640 1 -- 172.21.15.107:0/3806016347 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4568070380 msgr2=0x7f4568159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:39.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456ca6e640 1 --2- 172.21.15.107:0/3806016347 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4568070380 0x7f4568159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456ca6e640 1 -- 172.21.15.107:0/3806016347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4564009420 con 0x7f45680714f0 2023-12-04T05:40:39.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.430+0000 7f456d26f640 1 --2- 172.21.15.107:0/3806016347 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4568070380 0x7f4568159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:40:39.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.434+0000 7f456ca6e640 1 --2- 172.21.15.107:0/3806016347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45680714f0 0x7f4568159540 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4558000c10 tx=0x7f455801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:39.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.434+0000 7f455dffb640 1 -- 172.21.15.107:0/3806016347 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45580bf070 con 0x7f45680714f0 2023-12-04T05:40:39.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.434+0000 7f455dffb640 1 -- 172.21.15.107:0/3806016347 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45580b1be0 con 0x7f45680714f0 2023-12-04T05:40:39.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.434+0000 7f455dffb640 1 -- 172.21.15.107:0/3806016347 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45580b9410 con 0x7f45680714f0 2023-12-04T05:40:39.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.434+0000 7f456f4d3640 1 -- 172.21.15.107:0/3806016347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45580094d0 con 0x7f45680714f0 2023-12-04T05:40:39.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.434+0000 7f456f4d3640 1 -- 172.21.15.107:0/3806016347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4568158240 con 0x7f45680714f0 2023-12-04T05:40:39.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.434+0000 7f456f4d3640 1 -- 172.21.15.107:0/3806016347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4568067870 con 0x7f45680714f0 2023-12-04T05:40:39.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.434+0000 7f455dffb640 1 -- 172.21.15.107:0/3806016347 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f45580b7040 con 0x7f45680714f0 2023-12-04T05:40:39.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.434+0000 7f455dffb640 1 --2- 172.21.15.107:0/3806016347 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f454c0668d0 0x7f454c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:39.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.434+0000 7f455dffb640 1 -- 172.21.15.107:0/3806016347 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45581218e0 con 0x7f45680714f0 2023-12-04T05:40:39.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.434+0000 7f456d26f640 1 --2- 172.21.15.107:0/3806016347 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f454c0668d0 0x7f454c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:39.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.434+0000 7f456d26f640 1 --2- 172.21.15.107:0/3806016347 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f454c0668d0 0x7f454c068d90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4564009ca0 tx=0x7f4564008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:39.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.438+0000 7f455dffb640 1 -- 172.21.15.107:0/3806016347 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f45580eeb20 con 0x7f45680714f0 2023-12-04T05:40:39.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.594+0000 7f456f4d3640 1 -- 172.21.15.107:0/3806016347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f45680c26d0 con 0x7f45680714f0 2023-12-04T05:40:39.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.598+0000 7f455dffb640 1 -- 172.21.15.107:0/3806016347 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f45580b5020 con 0x7f45680714f0 2023-12-04T05:40:39.600 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-12-04T05:40:39.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.598+0000 7f456f4d3640 1 -- 172.21.15.107:0/3806016347 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f454c0668d0 msgr2=0x7f454c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:39.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.598+0000 7f456f4d3640 1 --2- 172.21.15.107:0/3806016347 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f454c0668d0 0x7f454c068d90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4564009ca0 tx=0x7f4564008040 comp rx=0 tx=0).stop 2023-12-04T05:40:39.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.598+0000 7f456f4d3640 1 -- 172.21.15.107:0/3806016347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45680714f0 msgr2=0x7f4568159540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:39.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.598+0000 7f456f4d3640 1 --2- 172.21.15.107:0/3806016347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45680714f0 0x7f4568159540 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4558000c10 tx=0x7f455801bad0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.598+0000 7f456f4d3640 1 -- 172.21.15.107:0/3806016347 shutdown_connections 2023-12-04T05:40:39.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.598+0000 7f456f4d3640 1 --2- 172.21.15.107:0/3806016347 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f454c0668d0 0x7f454c068d90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.598+0000 7f456f4d3640 1 --2- 172.21.15.107:0/3806016347 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4568151180 0x7f456815a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.598+0000 7f456f4d3640 1 --2- 172.21.15.107:0/3806016347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45680714f0 0x7f4568159540 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.598+0000 7f456f4d3640 1 --2- 172.21.15.107:0/3806016347 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4568070380 0x7f4568159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.598+0000 7f456f4d3640 1 -- 172.21.15.107:0/3806016347 >> 172.21.15.107:0/3806016347 conn(0x7f45680c7ff0 msgr2=0x7f456814e990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:39.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.598+0000 7f456f4d3640 1 -- 172.21.15.107:0/3806016347 shutdown_connections 2023-12-04T05:40:39.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.598+0000 7f456f4d3640 1 -- 172.21.15.107:0/3806016347 wait complete. 2023-12-04T05:40:39.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-04T05:40:39.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.738+0000 7fe88b0b7640 1 Processor -- start 2023-12-04T05:40:39.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.738+0000 7fe88b0b7640 1 -- start start 2023-12-04T05:40:39.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.738+0000 7fe88b0b7640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe884154020 0x7fe88415a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:39.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.738+0000 7fe88b0b7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe8840c4c10 0x7fe8840c5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:39.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.738+0000 7fe88b0b7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe88415ad50 con 0x7fe8840d01c0 2023-12-04T05:40:39.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.738+0000 7fe88b0b7640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8840ca230 con 0x7fe8840c4c10 2023-12-04T05:40:39.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.738+0000 7fe88b0b7640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8840c6770 con 0x7fe884154020 2023-12-04T05:40:39.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.738+0000 7fe889654640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe8840c4c10 0x7fe8840c5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:39.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.738+0000 7fe883fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe8840d01c0 0x7fe884071ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39588/0 (socket says 172.21.15.107:39588) 2023-12-04T05:40:39.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.738+0000 7fe889654640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe8840c4c10 0x7fe8840c5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56172/0 (socket says 172.21.15.107:56172) 2023-12-04T05:40:39.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.738+0000 7fe883fff640 1 -- 172.21.15.107:0/3021137823 learned_addr learned my addr 172.21.15.107:0/3021137823 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:39.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.738+0000 7fe888e53640 1 --2- 172.21.15.107:0/3021137823 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe884154020 0x7fe88415a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:39.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.738+0000 7fe889654640 1 -- 172.21.15.107:0/3021137823 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe884154020 msgr2=0x7fe88415a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:39.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.738+0000 7fe889654640 1 --2- 172.21.15.107:0/3021137823 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe884154020 0x7fe88415a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.738+0000 7fe889654640 1 -- 172.21.15.107:0/3021137823 >> v1:172.21.15.107:6789/0 conn(0x7fe8840d01c0 legacy=0x7fe884071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:39.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.738+0000 7fe889654640 1 -- 172.21.15.107:0/3021137823 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8840c55c0 con 0x7fe8840c4c10 2023-12-04T05:40:39.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.738+0000 7fe889654640 1 --2- 172.21.15.107:0/3021137823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8840c4c10 0x7fe8840c5080 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009d90 tx=0x7fe87c01c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37be6a548d74b667 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:39.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.738+0000 7fe8837fe640 1 -- 172.21.15.107:0/3021137823 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c0a5890 con 0x7fe8840c4c10 2023-12-04T05:40:39.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.738+0000 7fe8837fe640 1 -- 172.21.15.107:0/3021137823 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87c0a59f0 con 0x7fe8840c4c10 2023-12-04T05:40:39.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.738+0000 7fe8837fe640 1 -- 172.21.15.107:0/3021137823 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c0a5d80 con 0x7fe8840c4c10 2023-12-04T05:40:39.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe88b0b7640 1 -- 172.21.15.107:0/3021137823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8840c4c10 msgr2=0x7fe8840c5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:39.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe88b0b7640 1 --2- 172.21.15.107:0/3021137823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8840c4c10 0x7fe8840c5080 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009d90 tx=0x7fe87c01c870 comp rx=0 tx=0).stop 2023-12-04T05:40:39.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe88b0b7640 1 -- 172.21.15.107:0/3021137823 shutdown_connections 2023-12-04T05:40:39.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe88b0b7640 1 --2- 172.21.15.107:0/3021137823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8840c4c10 0x7fe8840c5080 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe88b0b7640 1 --2- 172.21.15.107:0/3021137823 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe884154020 0x7fe88415a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe88b0b7640 1 -- 172.21.15.107:0/3021137823 >> 172.21.15.107:0/3021137823 conn(0x7fe88406c6c0 msgr2=0x7fe8840cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:39.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe88b0b7640 1 -- 172.21.15.107:0/3021137823 shutdown_connections 2023-12-04T05:40:39.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe88b0b7640 1 -- 172.21.15.107:0/3021137823 wait complete. 2023-12-04T05:40:39.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe88b0b7640 1 Processor -- start 2023-12-04T05:40:39.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe88b0b7640 1 -- start start 2023-12-04T05:40:39.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe88b0b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8840c4c10 0x7fe8840c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:39.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe88b0b7640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe8840d01c0 0x7fe884158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:39.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe88b0b7640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe884154020 0x7fe8841efa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:39.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe88b0b7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8841f3b80 con 0x7fe8840c4c10 2023-12-04T05:40:39.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe88b0b7640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8841f2f90 con 0x7fe8840d01c0 2023-12-04T05:40:39.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe88b0b7640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8840cf670 con 0x7fe884154020 2023-12-04T05:40:39.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe888e53640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8840c4c10 0x7fe8840c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:39.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe888e53640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8840c4c10 0x7fe8840c6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56184/0 (socket says 172.21.15.107:56184) 2023-12-04T05:40:39.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe888e53640 1 -- 172.21.15.107:0/3885735156 learned_addr learned my addr 172.21.15.107:0/3885735156 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:39.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe883fff640 1 --2- 172.21.15.107:0/3885735156 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe8840d01c0 0x7fe884158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:39.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe889654640 1 --2- 172.21.15.107:0/3885735156 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe884154020 0x7fe8841efa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:39.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe888e53640 1 -- 172.21.15.107:0/3885735156 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe884154020 msgr2=0x7fe8841efa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:39.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe888e53640 1 --2- 172.21.15.107:0/3885735156 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe884154020 0x7fe8841efa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe888e53640 1 -- 172.21.15.107:0/3885735156 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe8840d01c0 msgr2=0x7fe884158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:39.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe888e53640 1 --2- 172.21.15.107:0/3885735156 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe8840d01c0 0x7fe884158fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe888e53640 1 -- 172.21.15.107:0/3885735156 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87c009a70 con 0x7fe8840c4c10 2023-12-04T05:40:39.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe888e53640 1 --2- 172.21.15.107:0/3885735156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8840c4c10 0x7fe8840c6a90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe87400c7f0 tx=0x7fe87400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:39.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe8817fa640 1 -- 172.21.15.107:0/3885735156 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe874004220 con 0x7fe8840c4c10 2023-12-04T05:40:39.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe8817fa640 1 -- 172.21.15.107:0/3885735156 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8740043a0 con 0x7fe8840c4c10 2023-12-04T05:40:39.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe88b0b7640 1 -- 172.21.15.107:0/3885735156 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8840cce70 con 0x7fe8840c4c10 2023-12-04T05:40:39.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe8817fa640 1 -- 172.21.15.107:0/3885735156 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe874004510 con 0x7fe8840c4c10 2023-12-04T05:40:39.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.742+0000 7fe88b0b7640 1 -- 172.21.15.107:0/3885735156 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8841f34c0 con 0x7fe8840c4c10 2023-12-04T05:40:39.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.746+0000 7fe88b0b7640 1 -- 172.21.15.107:0/3885735156 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8841f3cc0 con 0x7fe8840c4c10 2023-12-04T05:40:39.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.746+0000 7fe8817fa640 1 -- 172.21.15.107:0/3885735156 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe874003a70 con 0x7fe8840c4c10 2023-12-04T05:40:39.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.746+0000 7fe8817fa640 1 --2- 172.21.15.107:0/3885735156 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe8700668d0 0x7fe870068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:39.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.746+0000 7fe883fff640 1 --2- 172.21.15.107:0/3885735156 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe8700668d0 0x7fe870068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:39.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.746+0000 7fe8817fa640 1 -- 172.21.15.107:0/3885735156 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe874084350 con 0x7fe8840c4c10 2023-12-04T05:40:39.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.746+0000 7fe883fff640 1 --2- 172.21.15.107:0/3885735156 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe8700668d0 0x7fe870068d90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe8840c9fe0 tx=0x7fe87800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:39.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.750+0000 7fe8817fa640 1 -- 172.21.15.107:0/3885735156 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe8841f3cc0 con 0x7fe8840c4c10 2023-12-04T05:40:39.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.890+0000 7fe88b0b7640 1 -- 172.21.15.107:0/3885735156 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe8841f3cc0 con 0x7fe8840c4c10 2023-12-04T05:40:39.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.894+0000 7fe8817fa640 1 -- 172.21.15.107:0/3885735156 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe8841f3cc0 con 0x7fe8840c4c10 2023-12-04T05:40:39.896 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-12-04T05:40:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.894+0000 7fe88b0b7640 1 -- 172.21.15.107:0/3885735156 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe8700668d0 msgr2=0x7fe870068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.894+0000 7fe88b0b7640 1 --2- 172.21.15.107:0/3885735156 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe8700668d0 0x7fe870068d90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe8840c9fe0 tx=0x7fe87800b040 comp rx=0 tx=0).stop 2023-12-04T05:40:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.894+0000 7fe88b0b7640 1 -- 172.21.15.107:0/3885735156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8840c4c10 msgr2=0x7fe8840c6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.894+0000 7fe88b0b7640 1 --2- 172.21.15.107:0/3885735156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8840c4c10 0x7fe8840c6a90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe87400c7f0 tx=0x7fe87400ccb0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.894+0000 7fe88b0b7640 1 -- 172.21.15.107:0/3885735156 shutdown_connections 2023-12-04T05:40:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.894+0000 7fe88b0b7640 1 --2- 172.21.15.107:0/3885735156 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe8700668d0 0x7fe870068d90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.894+0000 7fe88b0b7640 1 --2- 172.21.15.107:0/3885735156 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe884154020 0x7fe8841efa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.894+0000 7fe88b0b7640 1 --2- 172.21.15.107:0/3885735156 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe8840d01c0 0x7fe884158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.894+0000 7fe88b0b7640 1 --2- 172.21.15.107:0/3885735156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8840c4c10 0x7fe8840c6a90 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:39.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.894+0000 7fe88b0b7640 1 -- 172.21.15.107:0/3885735156 >> 172.21.15.107:0/3885735156 conn(0x7fe88406c6c0 msgr2=0x7fe8840cb4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:39.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.894+0000 7fe88b0b7640 1 -- 172.21.15.107:0/3885735156 shutdown_connections 2023-12-04T05:40:39.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:39.894+0000 7fe88b0b7640 1 -- 172.21.15.107:0/3885735156 wait complete. 2023-12-04T05:40:39.913 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-04T05:40:40.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.022+0000 7ff511028640 1 Processor -- start 2023-12-04T05:40:40.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.026+0000 7ff511028640 1 -- start start 2023-12-04T05:40:40.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.026+0000 7ff511028640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff50c0d01c0 0x7ff50c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.026+0000 7ff511028640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff50c15d710 con 0x7ff50c072220 2023-12-04T05:40:40.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.026+0000 7ff511028640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff50c15d9f0 con 0x7ff50c154020 2023-12-04T05:40:40.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.026+0000 7ff511028640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff50c0ca230 con 0x7ff50c0d01c0 2023-12-04T05:40:40.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.026+0000 7ff503fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff50c0d01c0 0x7ff50c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.026+0000 7ff50b59e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff50c072220 0x7ff50c0c4c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39598/0 (socket says 172.21.15.107:39598) 2023-12-04T05:40:40.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.026+0000 7ff503fff640 1 --2- 172.21.15.107:0/494781762 >> v2:172.21.15.107:3300/0 conn(0x7ff50c0d01c0 0x7ff50c071ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56198/0 (socket says 172.21.15.107:56198) 2023-12-04T05:40:40.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.026+0000 7ff50b59e640 1 -- 172.21.15.107:0/494781762 learned_addr learned my addr 172.21.15.107:0/494781762 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:40.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.026+0000 7ff503fff640 1 -- 172.21.15.107:0/494781762 >> v1:172.21.15.195:6789/0 conn(0x7ff50c154020 legacy=0x7ff50c15a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:40.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.026+0000 7ff503fff640 1 -- 172.21.15.107:0/494781762 >> v1:172.21.15.107:6789/0 conn(0x7ff50c072220 legacy=0x7ff50c0c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:40.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.026+0000 7ff503fff640 1 -- 172.21.15.107:0/494781762 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50c15dcd0 con 0x7ff50c0d01c0 2023-12-04T05:40:40.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.026+0000 7ff50a59c640 1 -- 172.21.15.107:0/494781762 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069927360 0 0) 0x7ff50c15d710 con 0x7ff50c072220 2023-12-04T05:40:40.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.026+0000 7ff503fff640 1 --2- 172.21.15.107:0/494781762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff50c0d01c0 0x7ff50c071ce0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff4f40097c0 tx=0x7ff4f401ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=888e2fb5bb860294 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:40.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.026+0000 7ff50a59c640 1 -- 172.21.15.107:0/494781762 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f40a8930 con 0x7ff50c0d01c0 2023-12-04T05:40:40.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.026+0000 7ff50a59c640 1 -- 172.21.15.107:0/494781762 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f40a8a90 con 0x7ff50c0d01c0 2023-12-04T05:40:40.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.026+0000 7ff50a59c640 1 -- 172.21.15.107:0/494781762 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f40a8e20 con 0x7ff50c0d01c0 2023-12-04T05:40:40.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff511028640 1 -- 172.21.15.107:0/494781762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff50c0d01c0 msgr2=0x7ff50c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff511028640 1 --2- 172.21.15.107:0/494781762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff50c0d01c0 0x7ff50c071ce0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff4f40097c0 tx=0x7ff4f401ec30 comp rx=0 tx=0).stop 2023-12-04T05:40:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff511028640 1 -- 172.21.15.107:0/494781762 shutdown_connections 2023-12-04T05:40:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff511028640 1 --2- 172.21.15.107:0/494781762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff50c0d01c0 0x7ff50c071ce0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff511028640 1 -- 172.21.15.107:0/494781762 >> 172.21.15.107:0/494781762 conn(0x7ff50c06c6c0 msgr2=0x7ff50c0cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff511028640 1 -- 172.21.15.107:0/494781762 shutdown_connections 2023-12-04T05:40:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff511028640 1 -- 172.21.15.107:0/494781762 wait complete. 2023-12-04T05:40:40.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff511028640 1 Processor -- start 2023-12-04T05:40:40.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff511028640 1 -- start start 2023-12-04T05:40:40.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff511028640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff50c072220 0x7ff50c14af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff511028640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff50c0d01c0 0x7ff50c0c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff511028640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff50c154020 0x7ff50c0b4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff511028640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c0c99a0 con 0x7ff50c072220 2023-12-04T05:40:40.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff511028640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c15cf00 con 0x7ff50c154020 2023-12-04T05:40:40.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff511028640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c15c370 con 0x7ff50c0d01c0 2023-12-04T05:40:40.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff50ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff50c072220 0x7ff50c14af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff50ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff50c072220 0x7ff50c14af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56206/0 (socket says 172.21.15.107:56206) 2023-12-04T05:40:40.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff50ad9d640 1 -- 172.21.15.107:0/2536233178 learned_addr learned my addr 172.21.15.107:0/2536233178 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:40.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff503fff640 1 --2- 172.21.15.107:0/2536233178 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff50c0d01c0 0x7ff50c0c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff50ad9d640 1 -- 172.21.15.107:0/2536233178 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff50c0d01c0 msgr2=0x7ff50c0c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:40.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff50ad9d640 1 --2- 172.21.15.107:0/2536233178 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff50c0d01c0 0x7ff50c0c9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff50b59e640 1 --2- 172.21.15.107:0/2536233178 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff50c154020 0x7ff50c0b4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff50ad9d640 1 -- 172.21.15.107:0/2536233178 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff50c154020 msgr2=0x7ff50c0b4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:40.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff50ad9d640 1 --2- 172.21.15.107:0/2536233178 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff50c154020 0x7ff50c0b4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff50ad9d640 1 -- 172.21.15.107:0/2536233178 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f4009420 con 0x7ff50c072220 2023-12-04T05:40:40.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff503fff640 1 --2- 172.21.15.107:0/2536233178 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff50c0d01c0 0x7ff50c0c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:40.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff50ad9d640 1 --2- 172.21.15.107:0/2536233178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff50c072220 0x7ff50c14af30 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00c450 tx=0x7ff4fc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:40.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff5037fe640 1 -- 172.21.15.107:0/2536233178 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc012830 con 0x7ff50c072220 2023-12-04T05:40:40.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff5037fe640 1 -- 172.21.15.107:0/2536233178 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4fc0129b0 con 0x7ff50c072220 2023-12-04T05:40:40.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff511028640 1 -- 172.21.15.107:0/2536233178 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff50c1ef2e0 con 0x7ff50c072220 2023-12-04T05:40:40.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff5037fe640 1 -- 172.21.15.107:0/2536233178 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc012d60 con 0x7ff50c072220 2023-12-04T05:40:40.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff511028640 1 -- 172.21.15.107:0/2536233178 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff50c15d390 con 0x7ff50c072220 2023-12-04T05:40:40.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.030+0000 7ff511028640 1 -- 172.21.15.107:0/2536233178 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff50c067870 con 0x7ff50c072220 2023-12-04T05:40:40.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.034+0000 7ff5037fe640 1 -- 172.21.15.107:0/2536233178 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff4fc01c460 con 0x7ff50c072220 2023-12-04T05:40:40.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.034+0000 7ff5037fe640 1 --2- 172.21.15.107:0/2536233178 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff4ec0668d0 0x7ff4ec068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.034+0000 7ff5037fe640 1 -- 172.21.15.107:0/2536233178 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff4fc0842c0 con 0x7ff50c072220 2023-12-04T05:40:40.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.034+0000 7ff503fff640 1 --2- 172.21.15.107:0/2536233178 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff4ec0668d0 0x7ff4ec068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.034+0000 7ff503fff640 1 --2- 172.21.15.107:0/2536233178 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff4ec0668d0 0x7ff4ec068d90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff4f40b2000 tx=0x7ff4f4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:40.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.038+0000 7ff5037fe640 1 -- 172.21.15.107:0/2536233178 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff4fc051510 con 0x7ff50c072220 2023-12-04T05:40:40.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.190+0000 7ff511028640 1 -- 172.21.15.107:0/2536233178 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff50c0c9bc0 con 0x7ff50c072220 2023-12-04T05:40:40.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.190+0000 7ff5037fe640 1 -- 172.21.15.107:0/2536233178 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff4fc054b30 con 0x7ff50c072220 2023-12-04T05:40:40.195 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-12-04T05:40:40.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.194+0000 7ff511028640 1 -- 172.21.15.107:0/2536233178 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff4ec0668d0 msgr2=0x7ff4ec068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:40.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.194+0000 7ff511028640 1 --2- 172.21.15.107:0/2536233178 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff4ec0668d0 0x7ff4ec068d90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff4f40b2000 tx=0x7ff4f4008040 comp rx=0 tx=0).stop 2023-12-04T05:40:40.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.194+0000 7ff511028640 1 -- 172.21.15.107:0/2536233178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff50c072220 msgr2=0x7ff50c14af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:40.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.194+0000 7ff511028640 1 --2- 172.21.15.107:0/2536233178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff50c072220 0x7ff50c14af30 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00c450 tx=0x7ff4fc00c910 comp rx=0 tx=0).stop 2023-12-04T05:40:40.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.194+0000 7ff511028640 1 -- 172.21.15.107:0/2536233178 shutdown_connections 2023-12-04T05:40:40.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.194+0000 7ff511028640 1 --2- 172.21.15.107:0/2536233178 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff4ec0668d0 0x7ff4ec068d90 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.194+0000 7ff511028640 1 --2- 172.21.15.107:0/2536233178 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff50c154020 0x7ff50c0b4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.194+0000 7ff511028640 1 --2- 172.21.15.107:0/2536233178 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff50c0d01c0 0x7ff50c0c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.194+0000 7ff511028640 1 --2- 172.21.15.107:0/2536233178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff50c072220 0x7ff50c14af30 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.194+0000 7ff511028640 1 -- 172.21.15.107:0/2536233178 >> 172.21.15.107:0/2536233178 conn(0x7ff50c06c6c0 msgr2=0x7ff50c1520b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:40.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.194+0000 7ff511028640 1 -- 172.21.15.107:0/2536233178 shutdown_connections 2023-12-04T05:40:40.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.194+0000 7ff511028640 1 -- 172.21.15.107:0/2536233178 wait complete. 2023-12-04T05:40:40.214 INFO:tasks.mds_thrash:mds thrasher using random seed: 1701668440 2023-12-04T05:40:40.214 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T05:40:40.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.330+0000 7f9394341640 1 Processor -- start 2023-12-04T05:40:40.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.330+0000 7f9394341640 1 -- start start 2023-12-04T05:40:40.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.334+0000 7f9394341640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f938c0c46a0 0x7f938c0c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.334+0000 7f9394341640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f938c157180 0x7f938c0c7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.334+0000 7f9394341640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f938c0c4a80 0x7f938c0c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.334+0000 7f9394341640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f938c156490 con 0x7f938c0c46a0 2023-12-04T05:40:40.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.334+0000 7f9394341640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f938c06e660 con 0x7f938c157180 2023-12-04T05:40:40.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.334+0000 7f9394341640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f938c156cf0 con 0x7f938c0c4a80 2023-12-04T05:40:40.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.334+0000 7f93920dd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f938c0c46a0 0x7f938c0c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.334+0000 7f93920dd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f938c0c46a0 0x7f938c0c8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56210/0 (socket says 172.21.15.107:56210) 2023-12-04T05:40:40.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.334+0000 7f93920dd640 1 -- 172.21.15.107:0/3067963604 learned_addr learned my addr 172.21.15.107:0/3067963604 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:40.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.334+0000 7f93918dc640 1 --2- 172.21.15.107:0/3067963604 >> v2:172.21.15.195:3300/0 conn(0x7f938c157180 0x7f938c0c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.334+0000 7f93928de640 1 --2- 172.21.15.107:0/3067963604 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f938c0c4a80 0x7f938c0c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.334+0000 7f93920dd640 1 -- 172.21.15.107:0/3067963604 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f938c0c4a80 msgr2=0x7f938c0c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:40.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.334+0000 7f93920dd640 1 --2- 172.21.15.107:0/3067963604 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f938c0c4a80 0x7f938c0c5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.334+0000 7f93920dd640 1 -- 172.21.15.107:0/3067963604 >> v2:172.21.15.195:3300/0 conn(0x7f938c157180 msgr2=0x7f938c0c7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:40.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.334+0000 7f93920dd640 1 --2- 172.21.15.107:0/3067963604 >> v2:172.21.15.195:3300/0 conn(0x7f938c157180 0x7f938c0c7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.334+0000 7f93920dd640 1 -- 172.21.15.107:0/3067963604 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938c06e0a0 con 0x7f938c0c46a0 2023-12-04T05:40:40.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.334+0000 7f93920dd640 1 --2- 172.21.15.107:0/3067963604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c0c46a0 0x7f938c0c8b10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f938c074700 tx=0x7f938801d610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7828a68f88fd577a server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:40.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.334+0000 7f93910db640 1 -- 172.21.15.107:0/3067963604 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93880ab610 con 0x7f938c0c46a0 2023-12-04T05:40:40.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.334+0000 7f93910db640 1 -- 172.21.15.107:0/3067963604 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93880a7040 con 0x7f938c0c46a0 2023-12-04T05:40:40.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f93910db640 1 -- 172.21.15.107:0/3067963604 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93880ab990 con 0x7f938c0c46a0 2023-12-04T05:40:40.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f9394341640 1 -- 172.21.15.107:0/3067963604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c0c46a0 msgr2=0x7f938c0c8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:40.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f9394341640 1 --2- 172.21.15.107:0/3067963604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c0c46a0 0x7f938c0c8b10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f938c074700 tx=0x7f938801d610 comp rx=0 tx=0).stop 2023-12-04T05:40:40.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f9394341640 1 -- 172.21.15.107:0/3067963604 shutdown_connections 2023-12-04T05:40:40.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f9394341640 1 --2- 172.21.15.107:0/3067963604 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f938c0c4a80 0x7f938c0c5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f9394341640 1 --2- 172.21.15.107:0/3067963604 >> v2:172.21.15.195:3300/0 conn(0x7f938c157180 0x7f938c0c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f9394341640 1 --2- 172.21.15.107:0/3067963604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c0c46a0 0x7f938c0c8b10 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f9394341640 1 -- 172.21.15.107:0/3067963604 >> 172.21.15.107:0/3067963604 conn(0x7f938c0c7ff0 msgr2=0x7f938c0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:40.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f9394341640 1 -- 172.21.15.107:0/3067963604 shutdown_connections 2023-12-04T05:40:40.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f9394341640 1 -- 172.21.15.107:0/3067963604 wait complete. 2023-12-04T05:40:40.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f9394341640 1 Processor -- start 2023-12-04T05:40:40.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f9394341640 1 -- start start 2023-12-04T05:40:40.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f9394341640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f938c0c46a0 0x7f938c1464c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f9394341640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f938c0c4a80 0x7f938c147da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f9394341640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c157180 0x7f938c1f3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f9394341640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938c1489a0 con 0x7f938c157180 2023-12-04T05:40:40.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f9394341640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938c1f0490 con 0x7f938c0c4a80 2023-12-04T05:40:40.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f9394341640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f938c1483e0 con 0x7f938c0c46a0 2023-12-04T05:40:40.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f93928de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c157180 0x7f938c1f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f93928de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c157180 0x7f938c1f3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56216/0 (socket says 172.21.15.107:56216) 2023-12-04T05:40:40.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f93928de640 1 -- 172.21.15.107:0/416221148 learned_addr learned my addr 172.21.15.107:0/416221148 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:40.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f93920dd640 1 --2- 172.21.15.107:0/416221148 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f938c0c46a0 0x7f938c1464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f93918dc640 1 --2- 172.21.15.107:0/416221148 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f938c0c4a80 0x7f938c147da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f93928de640 1 -- 172.21.15.107:0/416221148 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f938c0c46a0 msgr2=0x7f938c1464c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:40.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f93928de640 1 --2- 172.21.15.107:0/416221148 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f938c0c46a0 0x7f938c1464c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f93928de640 1 -- 172.21.15.107:0/416221148 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f938c0c4a80 msgr2=0x7f938c147da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:40.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f93928de640 1 --2- 172.21.15.107:0/416221148 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f938c0c4a80 0x7f938c147da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f93928de640 1 -- 172.21.15.107:0/416221148 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93880094d0 con 0x7f938c157180 2023-12-04T05:40:40.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f93920dd640 1 --2- 172.21.15.107:0/416221148 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f938c0c46a0 0x7f938c1464c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:40.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f93928de640 1 --2- 172.21.15.107:0/416221148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c157180 0x7f938c1f3860 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9384007e60 tx=0x7f938400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:40.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f9382ffd640 1 -- 172.21.15.107:0/416221148 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9384014620 con 0x7f938c157180 2023-12-04T05:40:40.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f9382ffd640 1 -- 172.21.15.107:0/416221148 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938400ee90 con 0x7f938c157180 2023-12-04T05:40:40.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f9382ffd640 1 -- 172.21.15.107:0/416221148 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93840149e0 con 0x7f938c157180 2023-12-04T05:40:40.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f9394341640 1 -- 172.21.15.107:0/416221148 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f938c1486c0 con 0x7f938c157180 2023-12-04T05:40:40.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.338+0000 7f9394341640 1 -- 172.21.15.107:0/416221148 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f938c1f0a10 con 0x7f938c157180 2023-12-04T05:40:40.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.342+0000 7f9394341640 1 -- 172.21.15.107:0/416221148 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f938c1f02f0 con 0x7f938c157180 2023-12-04T05:40:40.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.342+0000 7f9382ffd640 1 -- 172.21.15.107:0/416221148 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9384014b40 con 0x7f938c157180 2023-12-04T05:40:40.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.342+0000 7f9382ffd640 1 --2- 172.21.15.107:0/416221148 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f93700668d0 0x7f9370068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.342+0000 7f93920dd640 1 --2- 172.21.15.107:0/416221148 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f93700668d0 0x7f9370068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.342+0000 7f9382ffd640 1 -- 172.21.15.107:0/416221148 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9384084dd0 con 0x7f938c157180 2023-12-04T05:40:40.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.342+0000 7f93920dd640 1 --2- 172.21.15.107:0/416221148 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f93700668d0 0x7f9370068d90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f93880af000 tx=0x7f9388008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:40.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.346+0000 7f9382ffd640 1 -- 172.21.15.107:0/416221148 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f938c1f02f0 con 0x7f938c157180 2023-12-04T05:40:40.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.506+0000 7f9394341640 1 -- 172.21.15.107:0/416221148 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f938c1f02f0 con 0x7f938c157180 2023-12-04T05:40:40.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.506+0000 7f9382ffd640 1 -- 172.21.15.107:0/416221148 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16790 (secure 0 0 0) 0x7f938c1f02f0 con 0x7f938c157180 2023-12-04T05:40:40.510 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:40.510 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":22,"fsid":"603e856b-31ac-475a-8f35-21ab7faadd48","created":"2023-12-04T05:39:34.921690+0000","modified":"2023-12-04T05:40:39.981913+0000","last_up_change":"2023-12-04T05:39:46.235790+0000","last_in_change":"2023-12-04T05:39:37.587639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T05:39:47.567702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T05:39:57.644286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T05:39:58.029897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ab7111c-0ae8-4d41-b6db-e3743614322c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6817","nonce":768714279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6821","nonce":768714279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6829","nonce":768714279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6825","nonce":768714279}]},"public_addr":"172.21.15.107:6817/768714279","cluster_addr":"172.21.15.107:6821/768714279","heartbeat_back_addr":"172.21.15.107:6829/768714279","heartbeat_front_addr":"172.21.15.107:6825/768714279","state":["exists","up"]},{"osd":1,"uuid":"baef7a1f-8026-4fab-9e70-48465271034e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6801","nonce":2496355981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6803","nonce":2496355981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6807","nonce":2496355981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6805","nonce":2496355981}]},"public_addr":"172.21.15.107:6801/2496355981","cluster_addr":"172.21.15.107:6803/2496355981","heartbeat_back_addr":"172.21.15.107:6807/2496355981","heartbeat_front_addr":"172.21.15.107:6805/2496355981","state":["exists","up"]},{"osd":2,"uuid":"90e8cf24-7b54-484a-9d85-b44a9d3dfeab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6809","nonce":3097298224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6811","nonce":3097298224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6815","nonce":3097298224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6813","nonce":3097298224}]},"public_addr":"172.21.15.107:6809/3097298224","cluster_addr":"172.21.15.107:6811/3097298224","heartbeat_back_addr":"172.21.15.107:6815/3097298224","heartbeat_front_addr":"172.21.15.107:6813/3097298224","state":["exists","up"]},{"osd":3,"uuid":"54658731-2b9a-4d5e-ab13-7c1be7d34e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6820","nonce":3773076671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6824","nonce":3773076671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6831","nonce":3773076671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6828","nonce":3773076671}]},"public_addr":"172.21.15.107:6820/3773076671","cluster_addr":"172.21.15.107:6824/3773076671","heartbeat_back_addr":"172.21.15.107:6831/3773076671","heartbeat_front_addr":"172.21.15.107:6828/3773076671","state":["exists","up"]},{"osd":4,"uuid":"9a9dea13-8afe-485d-aa43-45eda691179b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6801","nonce":2747823542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6803","nonce":2747823542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6807","nonce":2747823542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6805","nonce":2747823542}]},"public_addr":"172.21.15.195:6801/2747823542","cluster_addr":"172.21.15.195:6803/2747823542","heartbeat_back_addr":"172.21.15.195:6807/2747823542","heartbeat_front_addr":"172.21.15.195:6805/2747823542","state":["exists","up"]},{"osd":5,"uuid":"db840d83-5a7d-468a-8aa2-1997945847c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6825","nonce":1484264540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6827","nonce":1484264540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6831","nonce":1484264540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6829","nonce":1484264540}]},"public_addr":"172.21.15.195:6825/1484264540","cluster_addr":"172.21.15.195:6827/1484264540","heartbeat_back_addr":"172.21.15.195:6831/1484264540","heartbeat_front_addr":"172.21.15.195:6829/1484264540","state":["exists","up"]},{"osd":6,"uuid":"38b5f165-bf91-467e-a615-4b4784337689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6811","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6813","nonce":2638471412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6817","nonce":2638471412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6823","nonce":2638471412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6821","nonce":2638471412}]},"public_addr":"172.21.15.195:6813/2638471412","cluster_addr":"172.21.15.195:6817/2638471412","heartbeat_back_addr":"172.21.15.195:6823/2638471412","heartbeat_front_addr":"172.21.15.195:6821/2638471412","state":["exists","up"]},{"osd":7,"uuid":"e5b3c596-494b-4df3-9437-bc2d52168ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6809","nonce":768189614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6812","nonce":768189614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6820","nonce":768189614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6816","nonce":768189614}]},"public_addr":"172.21.15.195:6809/768189614","cluster_addr":"172.21.15.195:6812/768189614","heartbeat_back_addr":"172.21.15.195:6820/768189614","heartbeat_front_addr":"172.21.15.195:6816/768189614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.209077+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.086601+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.221568+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.238524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.059477+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.148422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.079867+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.064916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T05:40:40.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.506+0000 7f9394341640 1 -- 172.21.15.107:0/416221148 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f93700668d0 msgr2=0x7f9370068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:40.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.506+0000 7f9394341640 1 --2- 172.21.15.107:0/416221148 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f93700668d0 0x7f9370068d90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f93880af000 tx=0x7f9388008040 comp rx=0 tx=0).stop 2023-12-04T05:40:40.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.506+0000 7f9394341640 1 -- 172.21.15.107:0/416221148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c157180 msgr2=0x7f938c1f3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:40.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.506+0000 7f9394341640 1 --2- 172.21.15.107:0/416221148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c157180 0x7f938c1f3860 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9384007e60 tx=0x7f938400e4b0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.510+0000 7f9394341640 1 -- 172.21.15.107:0/416221148 shutdown_connections 2023-12-04T05:40:40.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.510+0000 7f9394341640 1 --2- 172.21.15.107:0/416221148 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f93700668d0 0x7f9370068d90 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.510+0000 7f9394341640 1 --2- 172.21.15.107:0/416221148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c157180 0x7f938c1f3860 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.510+0000 7f9394341640 1 --2- 172.21.15.107:0/416221148 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f938c0c4a80 0x7f938c147da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.510+0000 7f9394341640 1 --2- 172.21.15.107:0/416221148 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f938c0c46a0 0x7f938c1464c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.510+0000 7f9394341640 1 -- 172.21.15.107:0/416221148 >> 172.21.15.107:0/416221148 conn(0x7f938c0c7ff0 msgr2=0x7f938c153d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:40.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.510+0000 7f9394341640 1 -- 172.21.15.107:0/416221148 shutdown_connections 2023-12-04T05:40:40.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.510+0000 7f9394341640 1 -- 172.21.15.107:0/416221148 wait complete. 2023-12-04T05:40:40.528 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-04T05:39:47.567702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-04T05:39:57.644286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-04T05:39:58.029897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-04T05:40:40.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-04T05:40:40.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c8886640 1 Processor -- start 2023-12-04T05:40:40.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c8886640 1 -- start start 2023-12-04T05:40:40.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c8886640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa5c40cdc20 0x7fa5c4151960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c8886640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c40c9570 con 0x7fa5c40ca2d0 2023-12-04T05:40:40.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c8886640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c40c8c80 con 0x7fa5c40ca6b0 2023-12-04T05:40:40.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c8886640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c4073290 con 0x7fa5c40cdc20 2023-12-04T05:40:40.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa5c40ca2d0 0x7fa5c4072570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39602/0 (socket says 172.21.15.107:39602) 2023-12-04T05:40:40.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c259c640 1 -- 172.21.15.107:0/2983591320 learned_addr learned my addr 172.21.15.107:0/2983591320 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:40.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c2d9d640 1 --2- 172.21.15.107:0/2983591320 >> v2:172.21.15.195:3300/0 conn(0x7fa5c40cdc20 0x7fa5c4151960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c159a640 1 -- 172.21.15.107:0/2983591320 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147418753 0 0) 0x7fa5c40c9570 con 0x7fa5c40ca2d0 2023-12-04T05:40:40.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c159a640 1 -- 172.21.15.107:0/2983591320 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5ac003530 con 0x7fa5c40ca2d0 2023-12-04T05:40:40.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c159a640 1 -- 172.21.15.107:0/2983591320 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1730967647 0 0) 0x7fa5ac003530 con 0x7fa5c40ca2d0 2023-12-04T05:40:40.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c159a640 1 -- 172.21.15.107:0/2983591320 >> v2:172.21.15.195:3300/0 conn(0x7fa5c40cdc20 msgr2=0x7fa5c4151960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:40.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c159a640 1 --2- 172.21.15.107:0/2983591320 >> v2:172.21.15.195:3300/0 conn(0x7fa5c40cdc20 0x7fa5c4151960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c159a640 1 -- 172.21.15.107:0/2983591320 >> v1:172.21.15.195:6789/0 conn(0x7fa5c40ca6b0 legacy=0x7fa5c40cd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:40.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c159a640 1 -- 172.21.15.107:0/2983591320 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c4072fb0 con 0x7fa5c40ca2d0 2023-12-04T05:40:40.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c159a640 1 -- 172.21.15.107:0/2983591320 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7fa5b4001680 con 0x7fa5c40ca2d0 2023-12-04T05:40:40.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c159a640 1 -- 172.21.15.107:0/2983591320 >> v1:172.21.15.107:6789/0 conn(0x7fa5c40ca2d0 legacy=0x7fa5c4072570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:40.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c159a640 1 --2- 172.21.15.107:0/2983591320 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5ac0039b0 0x7fa5ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c159a640 1 --2- 172.21.15.107:0/2983591320 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5ac0047d0 0x7fa5ac004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c159a640 1 --2- 172.21.15.107:0/2983591320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5ac005190 0x7fa5ac00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c159a640 1 -- 172.21.15.107:0/2983591320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c4073290 con 0x7fa5ac005190 2023-12-04T05:40:40.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c159a640 1 -- 172.21.15.107:0/2983591320 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ac00c040 con 0x7fa5ac0039b0 2023-12-04T05:40:40.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c159a640 1 -- 172.21.15.107:0/2983591320 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ac00c200 con 0x7fa5ac0047d0 2023-12-04T05:40:40.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c2d9d640 1 --2- 172.21.15.107:0/2983591320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5ac005190 0x7fa5ac00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c259c640 1 --2- 172.21.15.107:0/2983591320 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5ac0039b0 0x7fa5ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c1d9b640 1 --2- 172.21.15.107:0/2983591320 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5ac0047d0 0x7fa5ac004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c2d9d640 1 -- 172.21.15.107:0/2983591320 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5ac0047d0 msgr2=0x7fa5ac004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:40.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c2d9d640 1 --2- 172.21.15.107:0/2983591320 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5ac0047d0 0x7fa5ac004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c2d9d640 1 -- 172.21.15.107:0/2983591320 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5ac0039b0 msgr2=0x7fa5ac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:40.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c2d9d640 1 --2- 172.21.15.107:0/2983591320 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5ac0039b0 0x7fa5ac003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.646+0000 7fa5c2d9d640 1 -- 172.21.15.107:0/2983591320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ac003530 con 0x7fa5ac005190 2023-12-04T05:40:40.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c2d9d640 1 -- 172.21.15.107:0/2983591320 reap_dead start 2023-12-04T05:40:40.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c2d9d640 1 --2- 172.21.15.107:0/2983591320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5ac005190 0x7fa5ac00b990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc009fc0 tx=0x7fa5bc01cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfb455b972022a13 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:40.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c159a640 1 -- 172.21.15.107:0/2983591320 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc0a5cb0 con 0x7fa5ac005190 2023-12-04T05:40:40.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c159a640 1 -- 172.21.15.107:0/2983591320 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5bc00f040 con 0x7fa5ac005190 2023-12-04T05:40:40.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c159a640 1 -- 172.21.15.107:0/2983591320 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc003be0 con 0x7fa5ac005190 2023-12-04T05:40:40.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c8886640 1 -- 172.21.15.107:0/2983591320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5ac005190 msgr2=0x7fa5ac00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:40.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c8886640 1 --2- 172.21.15.107:0/2983591320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5ac005190 0x7fa5ac00b990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc009fc0 tx=0x7fa5bc01cca0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c8886640 1 -- 172.21.15.107:0/2983591320 shutdown_connections 2023-12-04T05:40:40.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c8886640 1 --2- 172.21.15.107:0/2983591320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5ac005190 0x7fa5ac00b990 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c8886640 1 -- 172.21.15.107:0/2983591320 >> 172.21.15.107:0/2983591320 conn(0x7fa5c40663a0 msgr2=0x7fa5c40667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:40.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c8886640 1 -- 172.21.15.107:0/2983591320 shutdown_connections 2023-12-04T05:40:40.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c8886640 1 -- 172.21.15.107:0/2983591320 wait complete. 2023-12-04T05:40:40.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c8886640 1 Processor -- start 2023-12-04T05:40:40.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c8886640 1 -- start start 2023-12-04T05:40:40.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c8886640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5ac005190 0x7fa5c4156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c8886640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5c414f300 0x7fa5c4142d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c8886640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5c414f760 0x7fa5c4143290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c8886640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c4150ff0 con 0x7fa5ac005190 2023-12-04T05:40:40.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c8886640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c414c080 con 0x7fa5c414f760 2023-12-04T05:40:40.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c8886640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c41511d0 con 0x7fa5c414f300 2023-12-04T05:40:40.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5ac005190 0x7fa5c4156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5ac005190 0x7fa5c4156d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56230/0 (socket says 172.21.15.107:56230) 2023-12-04T05:40:40.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.650+0000 7fa5c259c640 1 -- 172.21.15.107:0/2867418831 learned_addr learned my addr 172.21.15.107:0/2867418831 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:40.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.654+0000 7fa5c1d9b640 1 --2- 172.21.15.107:0/2867418831 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5c414f300 0x7fa5c4142d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.654+0000 7fa5c2d9d640 1 --2- 172.21.15.107:0/2867418831 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5c414f760 0x7fa5c4143290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.654+0000 7fa5c259c640 1 -- 172.21.15.107:0/2867418831 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5c414f300 msgr2=0x7fa5c4142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:40.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.654+0000 7fa5c259c640 1 --2- 172.21.15.107:0/2867418831 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5c414f300 0x7fa5c4142d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.654+0000 7fa5c259c640 1 -- 172.21.15.107:0/2867418831 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5c414f760 msgr2=0x7fa5c4143290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:40.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.654+0000 7fa5c259c640 1 --2- 172.21.15.107:0/2867418831 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5c414f760 0x7fa5c4143290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.654+0000 7fa5c259c640 1 -- 172.21.15.107:0/2867418831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c41edf20 con 0x7fa5ac005190 2023-12-04T05:40:40.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.654+0000 7fa5c2d9d640 1 --2- 172.21.15.107:0/2867418831 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5c414f760 0x7fa5c4143290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:40.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.654+0000 7fa5c259c640 1 --2- 172.21.15.107:0/2867418831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5ac005190 0x7fa5c4156d50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa5b400e580 tx=0x7fa5b400ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:40.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.654+0000 7fa5b2ffd640 1 -- 172.21.15.107:0/2867418831 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b4012590 con 0x7fa5ac005190 2023-12-04T05:40:40.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.654+0000 7fa5b2ffd640 1 -- 172.21.15.107:0/2867418831 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b4012710 con 0x7fa5ac005190 2023-12-04T05:40:40.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.654+0000 7fa5c8886640 1 -- 172.21.15.107:0/2867418831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c41ee200 con 0x7fa5ac005190 2023-12-04T05:40:40.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.654+0000 7fa5b2ffd640 1 -- 172.21.15.107:0/2867418831 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b4012ac0 con 0x7fa5ac005190 2023-12-04T05:40:40.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.654+0000 7fa5c8886640 1 -- 172.21.15.107:0/2867418831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c41577c0 con 0x7fa5ac005190 2023-12-04T05:40:40.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.654+0000 7fa5c8886640 1 -- 172.21.15.107:0/2867418831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c41ee440 con 0x7fa5ac005190 2023-12-04T05:40:40.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.654+0000 7fa5b2ffd640 1 -- 172.21.15.107:0/2867418831 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa5b4012c20 con 0x7fa5ac005190 2023-12-04T05:40:40.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.654+0000 7fa5b2ffd640 1 --2- 172.21.15.107:0/2867418831 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa5a40668d0 0x7fa5a4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.654+0000 7fa5c1d9b640 1 --2- 172.21.15.107:0/2867418831 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa5a40668d0 0x7fa5a4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.654+0000 7fa5b2ffd640 1 -- 172.21.15.107:0/2867418831 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa5b4084380 con 0x7fa5ac005190 2023-12-04T05:40:40.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.654+0000 7fa5c1d9b640 1 --2- 172.21.15.107:0/2867418831 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa5a40668d0 0x7fa5a4068d90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa5c414c9f0 tx=0x7fa5b8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:40.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.658+0000 7fa5b2ffd640 1 -- 172.21.15.107:0/2867418831 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa5b4054eb0 con 0x7fa5ac005190 2023-12-04T05:40:40.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.810+0000 7fa5c8886640 1 -- 172.21.15.107:0/2867418831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa5c414be70 con 0x7fa5ac005190 2023-12-04T05:40:40.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.814+0000 7fa5b2ffd640 1 -- 172.21.15.107:0/2867418831 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa5b4012ed0 con 0x7fa5ac005190 2023-12-04T05:40:40.816 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-12-04T05:40:40.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.814+0000 7fa5c8886640 1 -- 172.21.15.107:0/2867418831 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa5a40668d0 msgr2=0x7fa5a4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:40.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.814+0000 7fa5c8886640 1 --2- 172.21.15.107:0/2867418831 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa5a40668d0 0x7fa5a4068d90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa5c414c9f0 tx=0x7fa5b8009040 comp rx=0 tx=0).stop 2023-12-04T05:40:40.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.814+0000 7fa5c8886640 1 -- 172.21.15.107:0/2867418831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5ac005190 msgr2=0x7fa5c4156d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:40.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.814+0000 7fa5c8886640 1 --2- 172.21.15.107:0/2867418831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5ac005190 0x7fa5c4156d50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa5b400e580 tx=0x7fa5b400ea40 comp rx=0 tx=0).stop 2023-12-04T05:40:40.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.814+0000 7fa5c8886640 1 -- 172.21.15.107:0/2867418831 shutdown_connections 2023-12-04T05:40:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.814+0000 7fa5c8886640 1 --2- 172.21.15.107:0/2867418831 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa5a40668d0 0x7fa5a4068d90 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.814+0000 7fa5c8886640 1 --2- 172.21.15.107:0/2867418831 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5c414f760 0x7fa5c4143290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.814+0000 7fa5c8886640 1 --2- 172.21.15.107:0/2867418831 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa5c414f300 0x7fa5c4142d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.814+0000 7fa5c8886640 1 --2- 172.21.15.107:0/2867418831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5ac005190 0x7fa5c4156d50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.814+0000 7fa5c8886640 1 -- 172.21.15.107:0/2867418831 >> 172.21.15.107:0/2867418831 conn(0x7fa5c40663a0 msgr2=0x7fa5c40c8040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.814+0000 7fa5c8886640 1 -- 172.21.15.107:0/2867418831 shutdown_connections 2023-12-04T05:40:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.814+0000 7fa5c8886640 1 -- 172.21.15.107:0/2867418831 wait complete. 2023-12-04T05:40:40.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-04T05:40:40.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1c6810640 1 Processor -- start 2023-12-04T05:40:40.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1c6810640 1 -- start start 2023-12-04T05:40:40.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1c6810640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff1c006b6c0 0x7ff1c006baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1c6810640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1c0072430 0x7ff1c015b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1c6810640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c015ac80 con 0x7ff1c015bdd0 2023-12-04T05:40:40.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1c6810640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1c006bfe0 con 0x7ff1c006b6c0 2023-12-04T05:40:40.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1c6810640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c015c260 con 0x7ff1c0072430 2023-12-04T05:40:40.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1c4dad640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff1c015bdd0 0x7ff1c015d340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39604/0 (socket says 172.21.15.107:39604) 2023-12-04T05:40:40.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1c4dad640 1 -- 172.21.15.107:0/1340506124 learned_addr learned my addr 172.21.15.107:0/1340506124 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:40.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1bf7fe640 1 --2- 172.21.15.107:0/1340506124 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1c0072430 0x7ff1c015b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1bffff640 1 --2- 172.21.15.107:0/1340506124 >> v2:172.21.15.195:3300/0 conn(0x7ff1c006b6c0 0x7ff1c006baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1beffd640 1 -- 172.21.15.107:0/1340506124 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194584427 0 0) 0x7ff1c015ac80 con 0x7ff1c015bdd0 2023-12-04T05:40:40.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1beffd640 1 -- 172.21.15.107:0/1340506124 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1a8003530 con 0x7ff1c015bdd0 2023-12-04T05:40:40.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1beffd640 1 -- 172.21.15.107:0/1340506124 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2067525537 0 0) 0x7ff1a8003530 con 0x7ff1c015bdd0 2023-12-04T05:40:40.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1beffd640 1 -- 172.21.15.107:0/1340506124 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1c0072430 msgr2=0x7ff1c015b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:40.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1beffd640 1 --2- 172.21.15.107:0/1340506124 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1c0072430 0x7ff1c015b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1beffd640 1 -- 172.21.15.107:0/1340506124 >> v2:172.21.15.195:3300/0 conn(0x7ff1c006b6c0 msgr2=0x7ff1c006baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:40.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1beffd640 1 --2- 172.21.15.107:0/1340506124 >> v2:172.21.15.195:3300/0 conn(0x7ff1c006b6c0 0x7ff1c006baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1beffd640 1 -- 172.21.15.107:0/1340506124 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c015a100 con 0x7ff1c015bdd0 2023-12-04T05:40:40.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1beffd640 1 -- 172.21.15.107:0/1340506124 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7ff1b8001a00 con 0x7ff1c015bdd0 2023-12-04T05:40:40.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1beffd640 1 -- 172.21.15.107:0/1340506124 >> v1:172.21.15.107:6789/0 conn(0x7ff1c015bdd0 legacy=0x7ff1c015d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:40.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1beffd640 1 --2- 172.21.15.107:0/1340506124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1a8003980 0x7ff1a8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1beffd640 1 --2- 172.21.15.107:0/1340506124 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1a80047d0 0x7ff1a8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1beffd640 1 --2- 172.21.15.107:0/1340506124 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1c0072430 0x7ff1a8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1beffd640 1 -- 172.21.15.107:0/1340506124 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c006bfe0 con 0x7ff1a8003980 2023-12-04T05:40:40.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1beffd640 1 -- 172.21.15.107:0/1340506124 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c015c260 con 0x7ff1a80047d0 2023-12-04T05:40:40.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1beffd640 1 -- 172.21.15.107:0/1340506124 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a800b990 con 0x7ff1c0072430 2023-12-04T05:40:40.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1bffff640 1 --2- 172.21.15.107:0/1340506124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1a8003980 0x7ff1a8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1c4dad640 1 --2- 172.21.15.107:0/1340506124 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1c0072430 0x7ff1a8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1bffff640 1 -- 172.21.15.107:0/1340506124 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1c0072430 msgr2=0x7ff1a8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:40.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1bffff640 1 --2- 172.21.15.107:0/1340506124 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1c0072430 0x7ff1a8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1bffff640 1 -- 172.21.15.107:0/1340506124 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1a80047d0 msgr2=0x7ff1a8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:40.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1bffff640 1 --2- 172.21.15.107:0/1340506124 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1a80047d0 0x7ff1a8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1bffff640 1 -- 172.21.15.107:0/1340506124 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a8003530 con 0x7ff1a8003980 2023-12-04T05:40:40.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.950+0000 7ff1c4dad640 1 --2- 172.21.15.107:0/1340506124 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1c0072430 0x7ff1a8005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:40.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1bffff640 1 --2- 172.21.15.107:0/1340506124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1a8003980 0x7ff1a8003da0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0009ae0 tx=0x7ff1b001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11244d0b07eb396a server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:40.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1beffd640 1 -- 172.21.15.107:0/1340506124 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b00b0400 con 0x7ff1a8003980 2023-12-04T05:40:40.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1beffd640 1 -- 172.21.15.107:0/1340506124 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b00b0560 con 0x7ff1a8003980 2023-12-04T05:40:40.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1beffd640 1 -- 172.21.15.107:0/1340506124 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b00b0910 con 0x7ff1a8003980 2023-12-04T05:40:40.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1c6810640 1 -- 172.21.15.107:0/1340506124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1a8003980 msgr2=0x7ff1a8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:40.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1c6810640 1 --2- 172.21.15.107:0/1340506124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1a8003980 0x7ff1a8003da0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0009ae0 tx=0x7ff1b001ca90 comp rx=0 tx=0).stop 2023-12-04T05:40:40.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1c4dad640 1 -- 172.21.15.107:0/1340506124 reap_dead start 2023-12-04T05:40:40.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1c6810640 1 -- 172.21.15.107:0/1340506124 shutdown_connections 2023-12-04T05:40:40.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1c6810640 1 -- 172.21.15.107:0/1340506124 >> 172.21.15.107:0/1340506124 conn(0x7ff1c00663a0 msgr2=0x7ff1c00667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:40.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1c6810640 1 -- 172.21.15.107:0/1340506124 shutdown_connections 2023-12-04T05:40:40.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1c6810640 1 -- 172.21.15.107:0/1340506124 wait complete. 2023-12-04T05:40:40.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1c6810640 1 Processor -- start 2023-12-04T05:40:40.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1c6810640 1 -- start start 2023-12-04T05:40:40.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1c6810640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1a8003980 0x7ff1c01511a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1c6810640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1c0157580 0x7ff1c015f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1c6810640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1c01f7410 0x7ff1c015f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1c6810640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c01f4320 con 0x7ff1a8003980 2023-12-04T05:40:40.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1c6810640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c01f3490 con 0x7ff1c01f7410 2023-12-04T05:40:40.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1c6810640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c01f7950 con 0x7ff1c0157580 2023-12-04T05:40:40.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1bffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1a8003980 0x7ff1c01511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1bffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1a8003980 0x7ff1c01511a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56250/0 (socket says 172.21.15.107:56250) 2023-12-04T05:40:40.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1bffff640 1 -- 172.21.15.107:0/2091283666 learned_addr learned my addr 172.21.15.107:0/2091283666 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:40.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1bffff640 1 -- 172.21.15.107:0/2091283666 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1c0157580 msgr2=0x7ff1c015f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:40.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1bffff640 1 --2- 172.21.15.107:0/2091283666 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1c0157580 0x7ff1c015f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1bffff640 1 -- 172.21.15.107:0/2091283666 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1c01f7410 msgr2=0x7ff1c015f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:40.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1bffff640 1 --2- 172.21.15.107:0/2091283666 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1c01f7410 0x7ff1c015f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:40.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1bffff640 1 -- 172.21.15.107:0/2091283666 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c01f36a0 con 0x7ff1a8003980 2023-12-04T05:40:40.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.954+0000 7ff1bffff640 1 --2- 172.21.15.107:0/2091283666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1a8003980 0x7ff1c01511a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff1b001c8c0 tx=0x7ff1b0019830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:40.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.958+0000 7ff1bcff9640 1 -- 172.21.15.107:0/2091283666 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b00b1d60 con 0x7ff1a8003980 2023-12-04T05:40:40.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.958+0000 7ff1bcff9640 1 -- 172.21.15.107:0/2091283666 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b00bb420 con 0x7ff1a8003980 2023-12-04T05:40:40.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.958+0000 7ff1c6810640 1 -- 172.21.15.107:0/2091283666 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c01f3980 con 0x7ff1a8003980 2023-12-04T05:40:40.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.958+0000 7ff1bcff9640 1 -- 172.21.15.107:0/2091283666 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b00bb790 con 0x7ff1a8003980 2023-12-04T05:40:40.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.958+0000 7ff1c6810640 1 -- 172.21.15.107:0/2091283666 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c01f3f40 con 0x7ff1a8003980 2023-12-04T05:40:40.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.958+0000 7ff1c6810640 1 -- 172.21.15.107:0/2091283666 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c0072c00 con 0x7ff1a8003980 2023-12-04T05:40:40.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.958+0000 7ff1bcff9640 1 -- 172.21.15.107:0/2091283666 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff1b00b9070 con 0x7ff1a8003980 2023-12-04T05:40:40.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.958+0000 7ff1bcff9640 1 --2- 172.21.15.107:0/2091283666 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff1ac066900 0x7ff1ac068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:40.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.958+0000 7ff1bcff9640 1 -- 172.21.15.107:0/2091283666 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff1b011e600 con 0x7ff1a8003980 2023-12-04T05:40:40.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.958+0000 7ff1bf7fe640 1 --2- 172.21.15.107:0/2091283666 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff1ac066900 0x7ff1ac068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:40.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.958+0000 7ff1bf7fe640 1 --2- 172.21.15.107:0/2091283666 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff1ac066900 0x7ff1ac068dc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4006aa0 tx=0x7ff1b4006a30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:40.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:40.962+0000 7ff1bcff9640 1 -- 172.21.15.107:0/2091283666 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff1b00eb7c0 con 0x7ff1a8003980 2023-12-04T05:40:41.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.118+0000 7ff1c6810640 1 -- 172.21.15.107:0/2091283666 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff1c0067870 con 0x7ff1a8003980 2023-12-04T05:40:41.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.118+0000 7ff1bcff9640 1 -- 172.21.15.107:0/2091283666 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff1b00eede0 con 0x7ff1a8003980 2023-12-04T05:40:41.123 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-12-04T05:40:41.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.122+0000 7ff1c6810640 1 -- 172.21.15.107:0/2091283666 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff1ac066900 msgr2=0x7ff1ac068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:41.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.122+0000 7ff1c6810640 1 --2- 172.21.15.107:0/2091283666 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff1ac066900 0x7ff1ac068dc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4006aa0 tx=0x7ff1b4006a30 comp rx=0 tx=0).stop 2023-12-04T05:40:41.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.122+0000 7ff1c6810640 1 -- 172.21.15.107:0/2091283666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1a8003980 msgr2=0x7ff1c01511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:41.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.122+0000 7ff1c6810640 1 --2- 172.21.15.107:0/2091283666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1a8003980 0x7ff1c01511a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff1b001c8c0 tx=0x7ff1b0019830 comp rx=0 tx=0).stop 2023-12-04T05:40:41.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.122+0000 7ff1c6810640 1 -- 172.21.15.107:0/2091283666 shutdown_connections 2023-12-04T05:40:41.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.122+0000 7ff1c6810640 1 --2- 172.21.15.107:0/2091283666 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff1ac066900 0x7ff1ac068dc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.122+0000 7ff1c6810640 1 --2- 172.21.15.107:0/2091283666 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff1c01f7410 0x7ff1c015f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.122+0000 7ff1c6810640 1 --2- 172.21.15.107:0/2091283666 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff1c0157580 0x7ff1c015f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.122+0000 7ff1c6810640 1 --2- 172.21.15.107:0/2091283666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1a8003980 0x7ff1c01511a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.122+0000 7ff1c6810640 1 -- 172.21.15.107:0/2091283666 >> 172.21.15.107:0/2091283666 conn(0x7ff1c00663a0 msgr2=0x7ff1c00cd750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:41.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.122+0000 7ff1c6810640 1 -- 172.21.15.107:0/2091283666 shutdown_connections 2023-12-04T05:40:41.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.122+0000 7ff1c6810640 1 -- 172.21.15.107:0/2091283666 wait complete. 2023-12-04T05:40:41.141 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-04T05:40:41.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.262+0000 7fdbcfc9c640 1 Processor -- start 2023-12-04T05:40:41.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.262+0000 7fdbcfc9c640 1 -- start start 2023-12-04T05:40:41.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.262+0000 7fdbcfc9c640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fdbc80724f0 0x7fdbc8158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:41.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.262+0000 7fdbcfc9c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdbc80c56f0 0x7fdbc8140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:41.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.262+0000 7fdbcfc9c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc8158da0 con 0x7fdbc8072110 2023-12-04T05:40:41.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.262+0000 7fdbcfc9c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbc81583d0 con 0x7fdbc80c56f0 2023-12-04T05:40:41.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.262+0000 7fdbcfc9c640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc80c6170 con 0x7fdbc80724f0 2023-12-04T05:40:41.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.262+0000 7fdbcda38640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdbc8072110 0x7fdbc80c7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39612/0 (socket says 172.21.15.107:39612) 2023-12-04T05:40:41.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.262+0000 7fdbcda38640 1 -- 172.21.15.107:0/3369497974 learned_addr learned my addr 172.21.15.107:0/3369497974 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:41.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.262+0000 7fdbce239640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdbc80c56f0 0x7fdbc8140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:41.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.262+0000 7fdbcd237640 1 --2- 172.21.15.107:0/3369497974 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fdbc80724f0 0x7fdbc8158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:41.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.262+0000 7fdbce239640 1 -- 172.21.15.107:0/3369497974 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fdbc80724f0 msgr2=0x7fdbc8158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:41.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.262+0000 7fdbce239640 1 --2- 172.21.15.107:0/3369497974 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fdbc80724f0 0x7fdbc8158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.262+0000 7fdbce239640 1 -- 172.21.15.107:0/3369497974 >> v1:172.21.15.107:6789/0 conn(0x7fdbc8072110 legacy=0x7fdbc80c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.262+0000 7fdbce239640 1 -- 172.21.15.107:0/3369497974 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc8157da0 con 0x7fdbc80c56f0 2023-12-04T05:40:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbce239640 1 --2- 172.21.15.107:0/3369497974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbc80c56f0 0x7fdbc8140830 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdbc00097c0 tx=0x7fdbc001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18f47aa951708cbf server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcca36640 1 -- 172.21.15.107:0/3369497974 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc00a8970 con 0x7fdbc80c56f0 2023-12-04T05:40:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcca36640 1 -- 172.21.15.107:0/3369497974 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc00a8ad0 con 0x7fdbc80c56f0 2023-12-04T05:40:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcca36640 1 -- 172.21.15.107:0/3369497974 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc00a8e60 con 0x7fdbc80c56f0 2023-12-04T05:40:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcfc9c640 1 -- 172.21.15.107:0/3369497974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbc80c56f0 msgr2=0x7fdbc8140830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcfc9c640 1 --2- 172.21.15.107:0/3369497974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbc80c56f0 0x7fdbc8140830 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdbc00097c0 tx=0x7fdbc001ec70 comp rx=0 tx=0).stop 2023-12-04T05:40:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcfc9c640 1 -- 172.21.15.107:0/3369497974 shutdown_connections 2023-12-04T05:40:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcfc9c640 1 --2- 172.21.15.107:0/3369497974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbc80c56f0 0x7fdbc8140830 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcfc9c640 1 --2- 172.21.15.107:0/3369497974 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fdbc80724f0 0x7fdbc8158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcfc9c640 1 -- 172.21.15.107:0/3369497974 >> 172.21.15.107:0/3369497974 conn(0x7fdbc80c7ff0 msgr2=0x7fdbc80c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcfc9c640 1 -- 172.21.15.107:0/3369497974 shutdown_connections 2023-12-04T05:40:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcfc9c640 1 -- 172.21.15.107:0/3369497974 wait complete. 2023-12-04T05:40:41.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcfc9c640 1 Processor -- start 2023-12-04T05:40:41.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcfc9c640 1 -- start start 2023-12-04T05:40:41.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcfc9c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbc8072110 0x7fdbc814e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:41.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcfc9c640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fdbc80724f0 0x7fdbc815a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcfc9c640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdbc80c56f0 0x7fdbc80ca340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcfc9c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc81f00a0 con 0x7fdbc8072110 2023-12-04T05:40:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcfc9c640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc81ef520 con 0x7fdbc80c56f0 2023-12-04T05:40:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcfc9c640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc81ef800 con 0x7fdbc80724f0 2023-12-04T05:40:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcda38640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbc8072110 0x7fdbc814e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcda38640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbc8072110 0x7fdbc814e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56278/0 (socket says 172.21.15.107:56278) 2023-12-04T05:40:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcda38640 1 -- 172.21.15.107:0/3749828064 learned_addr learned my addr 172.21.15.107:0/3749828064 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbce239640 1 --2- 172.21.15.107:0/3749828064 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdbc80c56f0 0x7fdbc80ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcda38640 1 -- 172.21.15.107:0/3749828064 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fdbc80724f0 msgr2=0x7fdbc815a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcda38640 1 --2- 172.21.15.107:0/3749828064 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fdbc80724f0 0x7fdbc815a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcda38640 1 -- 172.21.15.107:0/3749828064 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdbc80c56f0 msgr2=0x7fdbc80ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcda38640 1 --2- 172.21.15.107:0/3749828064 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdbc80c56f0 0x7fdbc80ca340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.266+0000 7fdbcda38640 1 -- 172.21.15.107:0/3749828064 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc0009420 con 0x7fdbc8072110 2023-12-04T05:40:41.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.270+0000 7fdbcda38640 1 --2- 172.21.15.107:0/3749828064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbc8072110 0x7fdbc814e700 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400c400 tx=0x7fdbc400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:41.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.270+0000 7fdbba7fc640 1 -- 172.21.15.107:0/3749828064 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4005cf0 con 0x7fdbc8072110 2023-12-04T05:40:41.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.270+0000 7fdbba7fc640 1 -- 172.21.15.107:0/3749828064 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc4005e50 con 0x7fdbc8072110 2023-12-04T05:40:41.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.270+0000 7fdbcfc9c640 1 -- 172.21.15.107:0/3749828064 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc81efae0 con 0x7fdbc8072110 2023-12-04T05:40:41.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.270+0000 7fdbba7fc640 1 -- 172.21.15.107:0/3749828064 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc40072d0 con 0x7fdbc8072110 2023-12-04T05:40:41.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.270+0000 7fdbcfc9c640 1 -- 172.21.15.107:0/3749828064 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc814c950 con 0x7fdbc8072110 2023-12-04T05:40:41.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.270+0000 7fdbcfc9c640 1 -- 172.21.15.107:0/3749828064 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc8067870 con 0x7fdbc8072110 2023-12-04T05:40:41.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.270+0000 7fdbba7fc640 1 -- 172.21.15.107:0/3749828064 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fdbc4007430 con 0x7fdbc8072110 2023-12-04T05:40:41.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.270+0000 7fdbba7fc640 1 --2- 172.21.15.107:0/3749828064 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fdbac0668d0 0x7fdbac068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:41.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.270+0000 7fdbba7fc640 1 -- 172.21.15.107:0/3749828064 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdbc4083f60 con 0x7fdbc8072110 2023-12-04T05:40:41.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.270+0000 7fdbcd237640 1 --2- 172.21.15.107:0/3749828064 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fdbac0668d0 0x7fdbac068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:41.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.270+0000 7fdbcd237640 1 --2- 172.21.15.107:0/3749828064 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fdbac0668d0 0x7fdbac068d90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdbc814fe90 tx=0x7fdbbc0098d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:41.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.274+0000 7fdbba7fc640 1 -- 172.21.15.107:0/3749828064 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fdbc40512b0 con 0x7fdbc8072110 2023-12-04T05:40:41.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.426+0000 7fdbcfc9c640 1 -- 172.21.15.107:0/3749828064 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdbc8158d50 con 0x7fdbc8072110 2023-12-04T05:40:41.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.426+0000 7fdbba7fc640 1 -- 172.21.15.107:0/3749828064 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdbc4017020 con 0x7fdbc8072110 2023-12-04T05:40:41.431 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-12-04T05:40:41.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.430+0000 7fdbcfc9c640 1 -- 172.21.15.107:0/3749828064 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fdbac0668d0 msgr2=0x7fdbac068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:41.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.430+0000 7fdbcfc9c640 1 --2- 172.21.15.107:0/3749828064 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fdbac0668d0 0x7fdbac068d90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdbc814fe90 tx=0x7fdbbc0098d0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.430+0000 7fdbcfc9c640 1 -- 172.21.15.107:0/3749828064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbc8072110 msgr2=0x7fdbc814e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:41.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.430+0000 7fdbcfc9c640 1 --2- 172.21.15.107:0/3749828064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbc8072110 0x7fdbc814e700 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400c400 tx=0x7fdbc400c8c0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.430+0000 7fdbcfc9c640 1 -- 172.21.15.107:0/3749828064 shutdown_connections 2023-12-04T05:40:41.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.430+0000 7fdbcfc9c640 1 --2- 172.21.15.107:0/3749828064 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fdbac0668d0 0x7fdbac068d90 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.430+0000 7fdbcfc9c640 1 --2- 172.21.15.107:0/3749828064 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdbc80c56f0 0x7fdbc80ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.430+0000 7fdbcfc9c640 1 --2- 172.21.15.107:0/3749828064 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fdbc80724f0 0x7fdbc815a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.430+0000 7fdbcfc9c640 1 --2- 172.21.15.107:0/3749828064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbc8072110 0x7fdbc814e700 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.430+0000 7fdbcfc9c640 1 -- 172.21.15.107:0/3749828064 >> 172.21.15.107:0/3749828064 conn(0x7fdbc80c7ff0 msgr2=0x7fdbc80c71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:41.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.430+0000 7fdbcfc9c640 1 -- 172.21.15.107:0/3749828064 shutdown_connections 2023-12-04T05:40:41.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.430+0000 7fdbcfc9c640 1 -- 172.21.15.107:0/3749828064 wait complete. 2023-12-04T05:40:41.450 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2023-12-04T05:40:41.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:40:41.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.566+0000 7f7e3801c640 1 Processor -- start 2023-12-04T05:40:41.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.566+0000 7f7e3801c640 1 -- start start 2023-12-04T05:40:41.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.566+0000 7f7e3801c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7e30154020 0x7f7e3015a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:41.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.566+0000 7f7e3801c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e3015ad50 con 0x7f7e300c4c10 2023-12-04T05:40:41.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.566+0000 7f7e3801c640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e3015d710 con 0x7f7e300d01c0 2023-12-04T05:40:41.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.566+0000 7f7e3801c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e300ca230 con 0x7f7e30154020 2023-12-04T05:40:41.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.566+0000 7f7e35db8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7e30154020 0x7f7e3015a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:41.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.566+0000 7f7e365b9640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7e300c4c10 0x7f7e300c5010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39628/0 (socket says 172.21.15.107:39628) 2023-12-04T05:40:41.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.566+0000 7f7e365b9640 1 -- 172.21.15.107:0/2601505596 learned_addr learned my addr 172.21.15.107:0/2601505596 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:41.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.570+0000 7f7e35db8640 1 -- 172.21.15.107:0/2601505596 >> v1:172.21.15.195:6789/0 conn(0x7f7e300d01c0 legacy=0x7f7e30071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:41.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.570+0000 7f7e35db8640 1 -- 172.21.15.107:0/2601505596 >> v1:172.21.15.107:6789/0 conn(0x7f7e300c4c10 legacy=0x7f7e300c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:41.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.570+0000 7f7e34db6640 1 -- 172.21.15.107:0/2601505596 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92407246 0 0) 0x7f7e3015ad50 con 0x7f7e300c4c10 2023-12-04T05:40:41.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.570+0000 7f7e35db8640 1 -- 172.21.15.107:0/2601505596 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3015d9f0 con 0x7f7e30154020 2023-12-04T05:40:41.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.570+0000 7f7e35db8640 1 --2- 172.21.15.107:0/2601505596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e30154020 0x7f7e3015a810 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c0097c0 tx=0x7f7e2c01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52045b90701ec3ff server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:41.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.570+0000 7f7e34db6640 1 -- 172.21.15.107:0/2601505596 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c0ab600 con 0x7f7e30154020 2023-12-04T05:40:41.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.570+0000 7f7e34db6640 1 -- 172.21.15.107:0/2601505596 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2c0a7050 con 0x7f7e30154020 2023-12-04T05:40:41.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.570+0000 7f7e34db6640 1 -- 172.21.15.107:0/2601505596 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c0ab980 con 0x7f7e30154020 2023-12-04T05:40:41.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.570+0000 7f7e3801c640 1 -- 172.21.15.107:0/2601505596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e30154020 msgr2=0x7f7e3015a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:41.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.570+0000 7f7e3801c640 1 --2- 172.21.15.107:0/2601505596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e30154020 0x7f7e3015a810 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c0097c0 tx=0x7f7e2c01c440 comp rx=0 tx=0).stop 2023-12-04T05:40:41.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.570+0000 7f7e3801c640 1 -- 172.21.15.107:0/2601505596 shutdown_connections 2023-12-04T05:40:41.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.570+0000 7f7e3801c640 1 --2- 172.21.15.107:0/2601505596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e30154020 0x7f7e3015a810 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.570+0000 7f7e3801c640 1 -- 172.21.15.107:0/2601505596 >> 172.21.15.107:0/2601505596 conn(0x7f7e3006c6c0 msgr2=0x7f7e300cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:41.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.570+0000 7f7e3801c640 1 -- 172.21.15.107:0/2601505596 shutdown_connections 2023-12-04T05:40:41.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.570+0000 7f7e3801c640 1 -- 172.21.15.107:0/2601505596 wait complete. 2023-12-04T05:40:41.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e3801c640 1 Processor -- start 2023-12-04T05:40:41.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e3801c640 1 -- start start 2023-12-04T05:40:41.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e3801c640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e300c4c10 0x7f7e3015c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:41.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e3801c640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e300d01c0 0x7f7e300ce050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:41.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e3801c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e30154020 0x7f7e300ce590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:41.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e3801c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e300b3620 con 0x7f7e30154020 2023-12-04T05:40:41.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e3801c640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e300b38e0 con 0x7f7e300c4c10 2023-12-04T05:40:41.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e3801c640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e300b3bc0 con 0x7f7e300d01c0 2023-12-04T05:40:41.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e365b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e30154020 0x7f7e300ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:41.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e365b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e30154020 0x7f7e300ce590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56296/0 (socket says 172.21.15.107:56296) 2023-12-04T05:40:41.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e365b9640 1 -- 172.21.15.107:0/1554095506 learned_addr learned my addr 172.21.15.107:0/1554095506 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:41.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e35db8640 1 --2- 172.21.15.107:0/1554095506 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e300c4c10 0x7f7e3015c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:41.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e355b7640 1 --2- 172.21.15.107:0/1554095506 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e300d01c0 0x7f7e300ce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:41.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e365b9640 1 -- 172.21.15.107:0/1554095506 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e300d01c0 msgr2=0x7f7e300ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:41.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e365b9640 1 --2- 172.21.15.107:0/1554095506 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e300d01c0 0x7f7e300ce050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e365b9640 1 -- 172.21.15.107:0/1554095506 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e300c4c10 msgr2=0x7f7e3015c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:41.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e365b9640 1 --2- 172.21.15.107:0/1554095506 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e300c4c10 0x7f7e3015c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e365b9640 1 -- 172.21.15.107:0/1554095506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c009420 con 0x7f7e30154020 2023-12-04T05:40:41.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e365b9640 1 --2- 172.21.15.107:0/1554095506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e30154020 0x7f7e300ce590 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7e28001570 tx=0x7f7e2800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:41.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e267fc640 1 -- 172.21.15.107:0/1554095506 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2800a640 con 0x7f7e30154020 2023-12-04T05:40:41.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e267fc640 1 -- 172.21.15.107:0/1554095506 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2800a7c0 con 0x7f7e30154020 2023-12-04T05:40:41.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e3801c640 1 -- 172.21.15.107:0/1554095506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e300b3ea0 con 0x7f7e30154020 2023-12-04T05:40:41.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e267fc640 1 -- 172.21.15.107:0/1554095506 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2800ab70 con 0x7f7e30154020 2023-12-04T05:40:41.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e3801c640 1 -- 172.21.15.107:0/1554095506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e300cd850 con 0x7f7e30154020 2023-12-04T05:40:41.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.574+0000 7f7e3801c640 1 -- 172.21.15.107:0/1554095506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e300cce70 con 0x7f7e30154020 2023-12-04T05:40:41.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.578+0000 7f7e267fc640 1 -- 172.21.15.107:0/1554095506 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f7e2800acd0 con 0x7f7e30154020 2023-12-04T05:40:41.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.578+0000 7f7e267fc640 1 --2- 172.21.15.107:0/1554095506 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7e140668d0 0x7f7e14068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:41.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.578+0000 7f7e267fc640 1 -- 172.21.15.107:0/1554095506 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7e28083e10 con 0x7f7e30154020 2023-12-04T05:40:41.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.578+0000 7f7e35db8640 1 --2- 172.21.15.107:0/1554095506 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7e140668d0 0x7f7e14068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:41.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.578+0000 7f7e35db8640 1 --2- 172.21.15.107:0/1554095506 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7e140668d0 0x7f7e14068d90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c0ad000 tx=0x7f7e2c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:41.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.582+0000 7f7e267fc640 1 -- 172.21.15.107:0/1554095506 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7e28050f40 con 0x7f7e30154020 2023-12-04T05:40:41.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.754+0000 7f7e3801c640 1 -- 172.21.15.107:0/1554095506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e300ca3e0 con 0x7f7e30154020 2023-12-04T05:40:41.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.754+0000 7f7e267fc640 1 -- 172.21.15.107:0/1554095506 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6563 (secure 0 0 0) 0x7f7e28018070 con 0x7f7e30154020 2023-12-04T05:40:41.758 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:41.758 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:40:28.948661+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":10,"addr":"172.21.15.195:6836/3769196693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":3769196693},{"type":"v1","addr":"172.21.15.195:6836","nonce":3769196693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:40:41.758 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-12-04T05:40:41.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.754+0000 7f7e3801c640 1 -- 172.21.15.107:0/1554095506 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7e140668d0 msgr2=0x7f7e14068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:41.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.754+0000 7f7e3801c640 1 --2- 172.21.15.107:0/1554095506 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7e140668d0 0x7f7e14068d90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c0ad000 tx=0x7f7e2c008040 comp rx=0 tx=0).stop 2023-12-04T05:40:41.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.754+0000 7f7e3801c640 1 -- 172.21.15.107:0/1554095506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e30154020 msgr2=0x7f7e300ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:41.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.754+0000 7f7e3801c640 1 --2- 172.21.15.107:0/1554095506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e30154020 0x7f7e300ce590 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7e28001570 tx=0x7f7e2800cca0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.754+0000 7f7e3801c640 1 -- 172.21.15.107:0/1554095506 shutdown_connections 2023-12-04T05:40:41.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.754+0000 7f7e3801c640 1 --2- 172.21.15.107:0/1554095506 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7e140668d0 0x7f7e14068d90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.754+0000 7f7e3801c640 1 --2- 172.21.15.107:0/1554095506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e30154020 0x7f7e300ce590 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.754+0000 7f7e3801c640 1 --2- 172.21.15.107:0/1554095506 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7e300d01c0 0x7f7e300ce050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.754+0000 7f7e3801c640 1 --2- 172.21.15.107:0/1554095506 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7e300c4c10 0x7f7e3015c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.754+0000 7f7e3801c640 1 -- 172.21.15.107:0/1554095506 >> 172.21.15.107:0/1554095506 conn(0x7f7e3006c6c0 msgr2=0x7f7e3014fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:41.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.754+0000 7f7e3801c640 1 -- 172.21.15.107:0/1554095506 shutdown_connections 2023-12-04T05:40:41.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.758+0000 7f7e3801c640 1 -- 172.21.15.107:0/1554095506 wait complete. 2023-12-04T05:40:41.775 INFO:tasks.mds_thrash:Ready to start thrashing 2023-12-04T05:40:41.775 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2023-12-04T05:40:41.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-04T05:40:41.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.882+0000 7f62f8097640 1 Processor -- start 2023-12-04T05:40:41.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.882+0000 7f62f8097640 1 -- start start 2023-12-04T05:40:41.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f8097640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f62f0072110 0x7f62f0158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:41.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f8097640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f62f00c56f0 0x7f62f0140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:41.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f8097640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f0158da0 con 0x7f62f00724f0 2023-12-04T05:40:41.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f8097640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62f01583d0 con 0x7f62f00c56f0 2023-12-04T05:40:41.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f8097640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f00c6170 con 0x7f62f0072110 2023-12-04T05:40:41.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f6634640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f62f00c56f0 0x7f62f0140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:41.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f6634640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f62f00c56f0 0x7f62f0140830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56300/0 (socket says 172.21.15.107:56300) 2023-12-04T05:40:41.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f6634640 1 -- 172.21.15.107:0/314607772 learned_addr learned my addr 172.21.15.107:0/314607772 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:41.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f5e33640 1 --2- 172.21.15.107:0/314607772 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f62f0072110 0x7f62f0158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:41.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f6634640 1 -- 172.21.15.107:0/314607772 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f62f0072110 msgr2=0x7f62f0158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:41.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f6634640 1 --2- 172.21.15.107:0/314607772 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f62f0072110 0x7f62f0158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f6634640 1 -- 172.21.15.107:0/314607772 >> v1:172.21.15.195:6789/0 conn(0x7f62f00724f0 legacy=0x7f62f00c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:41.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f6634640 1 -- 172.21.15.107:0/314607772 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f0157da0 con 0x7f62f00c56f0 2023-12-04T05:40:41.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f5e33640 1 --2- 172.21.15.107:0/314607772 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f62f0072110 0x7f62f0158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:41.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f6634640 1 --2- 172.21.15.107:0/314607772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62f00c56f0 0x7f62f0140830 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f62e8009870 tx=0x7f62e801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9eac39640098ac server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:41.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f4e31640 1 -- 172.21.15.107:0/314607772 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e80ac410 con 0x7f62f00c56f0 2023-12-04T05:40:41.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f4e31640 1 -- 172.21.15.107:0/314607772 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e80ac570 con 0x7f62f00c56f0 2023-12-04T05:40:41.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f4e31640 1 -- 172.21.15.107:0/314607772 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e80ac900 con 0x7f62f00c56f0 2023-12-04T05:40:41.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f8097640 1 -- 172.21.15.107:0/314607772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62f00c56f0 msgr2=0x7f62f0140830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:41.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f8097640 1 --2- 172.21.15.107:0/314607772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62f00c56f0 0x7f62f0140830 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f62e8009870 tx=0x7f62e801c500 comp rx=0 tx=0).stop 2023-12-04T05:40:41.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f8097640 1 -- 172.21.15.107:0/314607772 shutdown_connections 2023-12-04T05:40:41.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f8097640 1 --2- 172.21.15.107:0/314607772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62f00c56f0 0x7f62f0140830 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f8097640 1 --2- 172.21.15.107:0/314607772 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f62f0072110 0x7f62f0158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f8097640 1 -- 172.21.15.107:0/314607772 >> 172.21.15.107:0/314607772 conn(0x7f62f00c7ff0 msgr2=0x7f62f00c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:41.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f8097640 1 -- 172.21.15.107:0/314607772 shutdown_connections 2023-12-04T05:40:41.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.886+0000 7f62f8097640 1 -- 172.21.15.107:0/314607772 wait complete. 2023-12-04T05:40:41.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62f8097640 1 Processor -- start 2023-12-04T05:40:41.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62f8097640 1 -- start start 2023-12-04T05:40:41.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62f8097640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f62f0072110 0x7f62f0135f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:41.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62f8097640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62f00724f0 0x7f62f006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:41.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62f8097640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f62f00c56f0 0x7f62f0151800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:41.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62f8097640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f0071c70 con 0x7f62f00724f0 2023-12-04T05:40:41.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62f8097640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f01364d0 con 0x7f62f0072110 2023-12-04T05:40:41.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62f8097640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f00701e0 con 0x7f62f00c56f0 2023-12-04T05:40:41.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62f5632640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62f00724f0 0x7f62f006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:41.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62f5632640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62f00724f0 0x7f62f006fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56314/0 (socket says 172.21.15.107:56314) 2023-12-04T05:40:41.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62f5632640 1 -- 172.21.15.107:0/3040645530 learned_addr learned my addr 172.21.15.107:0/3040645530 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:41.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62f5632640 1 -- 172.21.15.107:0/3040645530 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f62f00c56f0 msgr2=0x7f62f0151800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:41.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62f5632640 1 --2- 172.21.15.107:0/3040645530 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f62f00c56f0 0x7f62f0151800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62f5632640 1 -- 172.21.15.107:0/3040645530 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f62f0072110 msgr2=0x7f62f0135f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:41.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62f5632640 1 --2- 172.21.15.107:0/3040645530 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f62f0072110 0x7f62f0135f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:41.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62f5632640 1 -- 172.21.15.107:0/3040645530 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e80094d0 con 0x7f62f00724f0 2023-12-04T05:40:41.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62f5632640 1 --2- 172.21.15.107:0/3040645530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62f00724f0 0x7f62f006fca0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f62e400c410 tx=0x7f62e400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:41.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62ee7fc640 1 -- 172.21.15.107:0/3040645530 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e4011790 con 0x7f62f00724f0 2023-12-04T05:40:41.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62ee7fc640 1 -- 172.21.15.107:0/3040645530 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e40118f0 con 0x7f62f00724f0 2023-12-04T05:40:41.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62f8097640 1 -- 172.21.15.107:0/3040645530 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f0140a90 con 0x7f62f00724f0 2023-12-04T05:40:41.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62ee7fc640 1 -- 172.21.15.107:0/3040645530 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e4011ca0 con 0x7f62f00724f0 2023-12-04T05:40:41.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62f8097640 1 -- 172.21.15.107:0/3040645530 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f0071990 con 0x7f62f00724f0 2023-12-04T05:40:41.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62f8097640 1 -- 172.21.15.107:0/3040645530 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f0067870 con 0x7f62f00724f0 2023-12-04T05:40:41.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.890+0000 7f62ee7fc640 1 -- 172.21.15.107:0/3040645530 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f62e401c460 con 0x7f62f00724f0 2023-12-04T05:40:41.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.894+0000 7f62ee7fc640 1 --2- 172.21.15.107:0/3040645530 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f62dc0668d0 0x7f62dc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:41.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.894+0000 7f62f5e33640 1 --2- 172.21.15.107:0/3040645530 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f62dc0668d0 0x7f62dc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:41.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.894+0000 7f62ee7fc640 1 -- 172.21.15.107:0/3040645530 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f62e4084290 con 0x7f62f00724f0 2023-12-04T05:40:41.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.894+0000 7f62f5e33640 1 --2- 172.21.15.107:0/3040645530 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f62dc0668d0 0x7f62dc068d90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f62e0008ea0 tx=0x7f62e00083d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:41.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:41.898+0000 7f62ee7fc640 1 -- 172.21.15.107:0/3040645530 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f62e40513c0 con 0x7f62f00724f0 2023-12-04T05:40:42.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.038+0000 7f62f8097640 1 -- 172.21.15.107:0/3040645530 --> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f62f00c83d0 con 0x7f62dc0668d0 2023-12-04T05:40:42.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.050+0000 7f62ee7fc640 1 -- 172.21.15.107:0/3040645530 <== mgr.4104 v2:172.21.15.107:6832/2693048026 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264548 (secure 0 0 0) 0x7f62f00c83d0 con 0x7f62dc0668d0 2023-12-04T05:40:42.055 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:42.056 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-12-04T05:40:42.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.050+0000 7f62f8097640 1 -- 172.21.15.107:0/3040645530 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f62dc0668d0 msgr2=0x7f62dc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:42.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.050+0000 7f62f8097640 1 --2- 172.21.15.107:0/3040645530 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f62dc0668d0 0x7f62dc068d90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f62e0008ea0 tx=0x7f62e00083d0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.050+0000 7f62f8097640 1 -- 172.21.15.107:0/3040645530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62f00724f0 msgr2=0x7f62f006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:42.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.050+0000 7f62f8097640 1 --2- 172.21.15.107:0/3040645530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62f00724f0 0x7f62f006fca0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f62e400c410 tx=0x7f62e400c8d0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.050+0000 7f62f8097640 1 -- 172.21.15.107:0/3040645530 shutdown_connections 2023-12-04T05:40:42.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.050+0000 7f62f8097640 1 --2- 172.21.15.107:0/3040645530 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f62dc0668d0 0x7f62dc068d90 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.050+0000 7f62f8097640 1 --2- 172.21.15.107:0/3040645530 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f62f00c56f0 0x7f62f0151800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.050+0000 7f62f8097640 1 --2- 172.21.15.107:0/3040645530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62f00724f0 0x7f62f006fca0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.050+0000 7f62f8097640 1 --2- 172.21.15.107:0/3040645530 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f62f0072110 0x7f62f0135f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.050+0000 7f62f8097640 1 -- 172.21.15.107:0/3040645530 >> 172.21.15.107:0/3040645530 conn(0x7f62f00c7ff0 msgr2=0x7f62f006d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:42.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.054+0000 7f62f8097640 1 -- 172.21.15.107:0/3040645530 shutdown_connections 2023-12-04T05:40:42.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.054+0000 7f62f8097640 1 -- 172.21.15.107:0/3040645530 wait complete. 2023-12-04T05:40:42.071 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2023-12-04T05:40:41.504546+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598402,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":364,"num_write_kb":2478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":296,"ondisk_log_size":296,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235880,"kb_used_data":3112,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738840,"statfs":{"total":773094113280,"available":772852572160,"internally_reserved":0,"allocated":3186688,"data_stored":1801452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002767"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503577+0000","last_change":"2023-12-04T05:39:59.736570+0000","last_active":"2023-12-04T05:40:01.503577+0000","last_peered":"2023-12-04T05:40:01.503577+0000","last_clean":"2023-12-04T05:40:01.503577+0000","last_became_active":"2023-12-04T05:39:59.736000+0000","last_became_peered":"2023-12-04T05:39:59.736000+0000","last_unstale":"2023-12-04T05:40:01.503577+0000","last_undegraded":"2023-12-04T05:40:01.503577+0000","last_fullsized":"2023-12-04T05:40:01.503577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:34:15.458889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908781+0000","last_change":"2023-12-04T05:39:59.703955+0000","last_active":"2023-12-04T05:40:00.908781+0000","last_peered":"2023-12-04T05:40:00.908781+0000","last_clean":"2023-12-04T05:40:00.908781+0000","last_became_active":"2023-12-04T05:39:59.703032+0000","last_became_peered":"2023-12-04T05:39:59.703032+0000","last_unstale":"2023-12-04T05:40:00.908781+0000","last_undegraded":"2023-12-04T05:40:00.908781+0000","last_fullsized":"2023-12-04T05:40:00.908781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:22:09.010348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501353+0000","last_change":"2023-12-04T05:39:59.705340+0000","last_active":"2023-12-04T05:40:01.501353+0000","last_peered":"2023-12-04T05:40:01.501353+0000","last_clean":"2023-12-04T05:40:01.501353+0000","last_became_active":"2023-12-04T05:39:59.699479+0000","last_became_peered":"2023-12-04T05:39:59.699479+0000","last_unstale":"2023-12-04T05:40:01.501353+0000","last_undegraded":"2023-12-04T05:40:01.501353+0000","last_fullsized":"2023-12-04T05:40:01.501353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:12:00.122168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683148+0000","last_change":"2023-12-04T05:39:59.728698+0000","last_active":"2023-12-04T05:40:00.683148+0000","last_peered":"2023-12-04T05:40:00.683148+0000","last_clean":"2023-12-04T05:40:00.683148+0000","last_became_active":"2023-12-04T05:39:59.727914+0000","last_became_peered":"2023-12-04T05:39:59.727914+0000","last_unstale":"2023-12-04T05:40:00.683148+0000","last_undegraded":"2023-12-04T05:40:00.683148+0000","last_fullsized":"2023-12-04T05:40:00.683148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:39:17.269322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502651+0000","last_change":"2023-12-04T05:39:59.726320+0000","last_active":"2023-12-04T05:40:01.502651+0000","last_peered":"2023-12-04T05:40:01.502651+0000","last_clean":"2023-12-04T05:40:01.502651+0000","last_became_active":"2023-12-04T05:39:59.722627+0000","last_became_peered":"2023-12-04T05:39:59.722627+0000","last_unstale":"2023-12-04T05:40:01.502651+0000","last_undegraded":"2023-12-04T05:40:01.502651+0000","last_fullsized":"2023-12-04T05:40:01.502651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:40:48.940938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682860+0000","last_change":"2023-12-04T05:39:59.718363+0000","last_active":"2023-12-04T05:40:00.682860+0000","last_peered":"2023-12-04T05:40:00.682860+0000","last_clean":"2023-12-04T05:40:00.682860+0000","last_became_active":"2023-12-04T05:39:59.717475+0000","last_became_peered":"2023-12-04T05:39:59.717475+0000","last_unstale":"2023-12-04T05:40:00.682860+0000","last_undegraded":"2023-12-04T05:40:00.682860+0000","last_fullsized":"2023-12-04T05:40:00.682860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:02:56.684095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501353+0000","last_change":"2023-12-04T05:39:59.724035+0000","last_active":"2023-12-04T05:40:01.501353+0000","last_peered":"2023-12-04T05:40:01.501353+0000","last_clean":"2023-12-04T05:40:01.501353+0000","last_became_active":"2023-12-04T05:39:59.723277+0000","last_became_peered":"2023-12-04T05:39:59.723277+0000","last_unstale":"2023-12-04T05:40:01.501353+0000","last_undegraded":"2023-12-04T05:40:01.501353+0000","last_fullsized":"2023-12-04T05:40:01.501353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:34:54.379064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502157+0000","last_change":"2023-12-04T05:39:59.720028+0000","last_active":"2023-12-04T05:40:01.502157+0000","last_peered":"2023-12-04T05:40:01.502157+0000","last_clean":"2023-12-04T05:40:01.502157+0000","last_became_active":"2023-12-04T05:39:59.719236+0000","last_became_peered":"2023-12-04T05:39:59.719236+0000","last_unstale":"2023-12-04T05:40:01.502157+0000","last_undegraded":"2023-12-04T05:40:01.502157+0000","last_fullsized":"2023-12-04T05:40:01.502157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:23:39.909425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503754+0000","last_change":"2023-12-04T05:39:59.732301+0000","last_active":"2023-12-04T05:40:01.503754+0000","last_peered":"2023-12-04T05:40:01.503754+0000","last_clean":"2023-12-04T05:40:01.503754+0000","last_became_active":"2023-12-04T05:39:59.731561+0000","last_became_peered":"2023-12-04T05:39:59.731561+0000","last_unstale":"2023-12-04T05:40:01.503754+0000","last_undegraded":"2023-12-04T05:40:01.503754+0000","last_fullsized":"2023-12-04T05:40:01.503754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:49:40.179867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501688+0000","last_change":"2023-12-04T05:39:59.721633+0000","last_active":"2023-12-04T05:40:01.501688+0000","last_peered":"2023-12-04T05:40:01.501688+0000","last_clean":"2023-12-04T05:40:01.501688+0000","last_became_active":"2023-12-04T05:39:59.720846+0000","last_became_peered":"2023-12-04T05:39:59.720846+0000","last_unstale":"2023-12-04T05:40:01.501688+0000","last_undegraded":"2023-12-04T05:40:01.501688+0000","last_fullsized":"2023-12-04T05:40:01.501688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:44:34.893307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501835+0000","last_change":"2023-12-04T05:39:59.721173+0000","last_active":"2023-12-04T05:40:01.501835+0000","last_peered":"2023-12-04T05:40:01.501835+0000","last_clean":"2023-12-04T05:40:01.501835+0000","last_became_active":"2023-12-04T05:39:59.720371+0000","last_became_peered":"2023-12-04T05:39:59.720371+0000","last_unstale":"2023-12-04T05:40:01.501835+0000","last_undegraded":"2023-12-04T05:40:01.501835+0000","last_fullsized":"2023-12-04T05:40:01.501835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:12:23.683644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682272+0000","last_change":"2023-12-04T05:39:59.710269+0000","last_active":"2023-12-04T05:40:00.682272+0000","last_peered":"2023-12-04T05:40:00.682272+0000","last_clean":"2023-12-04T05:40:00.682272+0000","last_became_active":"2023-12-04T05:39:59.709565+0000","last_became_peered":"2023-12-04T05:39:59.709565+0000","last_unstale":"2023-12-04T05:40:00.682272+0000","last_undegraded":"2023-12-04T05:40:00.682272+0000","last_fullsized":"2023-12-04T05:40:00.682272+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:11:13.222803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501884+0000","last_change":"2023-12-04T05:39:59.710484+0000","last_active":"2023-12-04T05:40:01.501884+0000","last_peered":"2023-12-04T05:40:01.501884+0000","last_clean":"2023-12-04T05:40:01.501884+0000","last_became_active":"2023-12-04T05:39:59.709623+0000","last_became_peered":"2023-12-04T05:39:59.709623+0000","last_unstale":"2023-12-04T05:40:01.501884+0000","last_undegraded":"2023-12-04T05:40:01.501884+0000","last_fullsized":"2023-12-04T05:40:01.501884+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:40:05.852673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.500880+0000","last_change":"2023-12-04T05:39:59.715685+0000","last_active":"2023-12-04T05:40:01.500880+0000","last_peered":"2023-12-04T05:40:01.500880+0000","last_clean":"2023-12-04T05:40:01.500880+0000","last_became_active":"2023-12-04T05:39:59.714890+0000","last_became_peered":"2023-12-04T05:39:59.714890+0000","last_unstale":"2023-12-04T05:40:01.500880+0000","last_undegraded":"2023-12-04T05:40:01.500880+0000","last_fullsized":"2023-12-04T05:40:01.500880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:59:34.243902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908703+0000","last_change":"2023-12-04T05:39:59.688405+0000","last_active":"2023-12-04T05:40:00.908703+0000","last_peered":"2023-12-04T05:40:00.908703+0000","last_clean":"2023-12-04T05:40:00.908703+0000","last_became_active":"2023-12-04T05:39:59.687603+0000","last_became_peered":"2023-12-04T05:39:59.687603+0000","last_unstale":"2023-12-04T05:40:00.908703+0000","last_undegraded":"2023-12-04T05:40:00.908703+0000","last_fullsized":"2023-12-04T05:40:00.908703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:44:51.269794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681203+0000","last_change":"2023-12-04T05:39:59.718543+0000","last_active":"2023-12-04T05:40:00.681203+0000","last_peered":"2023-12-04T05:40:00.681203+0000","last_clean":"2023-12-04T05:40:00.681203+0000","last_became_active":"2023-12-04T05:39:59.717826+0000","last_became_peered":"2023-12-04T05:39:59.717826+0000","last_unstale":"2023-12-04T05:40:00.681203+0000","last_undegraded":"2023-12-04T05:40:00.681203+0000","last_fullsized":"2023-12-04T05:40:00.681203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:31:03.248114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.684087+0000","last_change":"2023-12-04T05:39:59.724278+0000","last_active":"2023-12-04T05:40:00.684087+0000","last_peered":"2023-12-04T05:40:00.684087+0000","last_clean":"2023-12-04T05:40:00.684087+0000","last_became_active":"2023-12-04T05:39:59.723384+0000","last_became_peered":"2023-12-04T05:39:59.723384+0000","last_unstale":"2023-12-04T05:40:00.684087+0000","last_undegraded":"2023-12-04T05:40:00.684087+0000","last_fullsized":"2023-12-04T05:40:00.684087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:07:20.576260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.679022+0000","last_change":"2023-12-04T05:39:59.711802+0000","last_active":"2023-12-04T05:40:00.679022+0000","last_peered":"2023-12-04T05:40:00.679022+0000","last_clean":"2023-12-04T05:40:00.679022+0000","last_became_active":"2023-12-04T05:39:59.711044+0000","last_became_peered":"2023-12-04T05:39:59.711044+0000","last_unstale":"2023-12-04T05:40:00.679022+0000","last_undegraded":"2023-12-04T05:40:00.679022+0000","last_fullsized":"2023-12-04T05:40:00.679022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:42:28.221010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503300+0000","last_change":"2023-12-04T05:39:59.725226+0000","last_active":"2023-12-04T05:40:01.503300+0000","last_peered":"2023-12-04T05:40:01.503300+0000","last_clean":"2023-12-04T05:40:01.503300+0000","last_became_active":"2023-12-04T05:39:59.724334+0000","last_became_peered":"2023-12-04T05:39:59.724334+0000","last_unstale":"2023-12-04T05:40:01.503300+0000","last_undegraded":"2023-12-04T05:40:01.503300+0000","last_fullsized":"2023-12-04T05:40:01.503300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:48:30.439285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501624+0000","last_change":"2023-12-04T05:39:59.729858+0000","last_active":"2023-12-04T05:40:01.501624+0000","last_peered":"2023-12-04T05:40:01.501624+0000","last_clean":"2023-12-04T05:40:01.501624+0000","last_became_active":"2023-12-04T05:39:59.728989+0000","last_became_peered":"2023-12-04T05:39:59.728989+0000","last_unstale":"2023-12-04T05:40:01.501624+0000","last_undegraded":"2023-12-04T05:40:01.501624+0000","last_fullsized":"2023-12-04T05:40:01.501624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:42:01.099743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501616+0000","last_change":"2023-12-04T05:39:59.724022+0000","last_active":"2023-12-04T05:40:01.501616+0000","last_peered":"2023-12-04T05:40:01.501616+0000","last_clean":"2023-12-04T05:40:01.501616+0000","last_became_active":"2023-12-04T05:39:59.723120+0000","last_became_peered":"2023-12-04T05:39:59.723120+0000","last_unstale":"2023-12-04T05:40:01.501616+0000","last_undegraded":"2023-12-04T05:40:01.501616+0000","last_fullsized":"2023-12-04T05:40:01.501616+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:13:50.048757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683322+0000","last_change":"2023-12-04T05:39:59.714806+0000","last_active":"2023-12-04T05:40:00.683322+0000","last_peered":"2023-12-04T05:40:00.683322+0000","last_clean":"2023-12-04T05:40:00.683322+0000","last_became_active":"2023-12-04T05:39:59.713917+0000","last_became_peered":"2023-12-04T05:39:59.713917+0000","last_unstale":"2023-12-04T05:40:00.683322+0000","last_undegraded":"2023-12-04T05:40:00.683322+0000","last_fullsized":"2023-12-04T05:40:00.683322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:18:03.128562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501966+0000","last_change":"2023-12-04T05:39:59.729098+0000","last_active":"2023-12-04T05:40:01.501966+0000","last_peered":"2023-12-04T05:40:01.501966+0000","last_clean":"2023-12-04T05:40:01.501966+0000","last_became_active":"2023-12-04T05:39:59.728449+0000","last_became_peered":"2023-12-04T05:39:59.728449+0000","last_unstale":"2023-12-04T05:40:01.501966+0000","last_undegraded":"2023-12-04T05:40:01.501966+0000","last_fullsized":"2023-12-04T05:40:01.501966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:25:12.098431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682551+0000","last_change":"2023-12-04T05:39:59.708300+0000","last_active":"2023-12-04T05:40:00.682551+0000","last_peered":"2023-12-04T05:40:00.682551+0000","last_clean":"2023-12-04T05:40:00.682551+0000","last_became_active":"2023-12-04T05:39:59.707530+0000","last_became_peered":"2023-12-04T05:39:59.707530+0000","last_unstale":"2023-12-04T05:40:00.682551+0000","last_undegraded":"2023-12-04T05:40:00.682551+0000","last_fullsized":"2023-12-04T05:40:00.682551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:11:40.053312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502562+0000","last_change":"2023-12-04T05:39:59.729406+0000","last_active":"2023-12-04T05:40:01.502562+0000","last_peered":"2023-12-04T05:40:01.502562+0000","last_clean":"2023-12-04T05:40:01.502562+0000","last_became_active":"2023-12-04T05:39:59.728677+0000","last_became_peered":"2023-12-04T05:39:59.728677+0000","last_unstale":"2023-12-04T05:40:01.502562+0000","last_undegraded":"2023-12-04T05:40:01.502562+0000","last_fullsized":"2023-12-04T05:40:01.502562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:03:26.954387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501922+0000","last_change":"2023-12-04T05:39:59.733628+0000","last_active":"2023-12-04T05:40:01.501922+0000","last_peered":"2023-12-04T05:40:01.501922+0000","last_clean":"2023-12-04T05:40:01.501922+0000","last_became_active":"2023-12-04T05:39:59.726137+0000","last_became_peered":"2023-12-04T05:39:59.726137+0000","last_unstale":"2023-12-04T05:40:01.501922+0000","last_undegraded":"2023-12-04T05:40:01.501922+0000","last_fullsized":"2023-12-04T05:40:01.501922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:49:17.630085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908788+0000","last_change":"2023-12-04T05:39:59.727169+0000","last_active":"2023-12-04T05:40:00.908788+0000","last_peered":"2023-12-04T05:40:00.908788+0000","last_clean":"2023-12-04T05:40:00.908788+0000","last_became_active":"2023-12-04T05:39:59.725378+0000","last_became_peered":"2023-12-04T05:39:59.725378+0000","last_unstale":"2023-12-04T05:40:00.908788+0000","last_undegraded":"2023-12-04T05:40:00.908788+0000","last_fullsized":"2023-12-04T05:40:00.908788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:23:21.007397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502427+0000","last_change":"2023-12-04T05:39:59.729516+0000","last_active":"2023-12-04T05:40:01.502427+0000","last_peered":"2023-12-04T05:40:01.502427+0000","last_clean":"2023-12-04T05:40:01.502427+0000","last_became_active":"2023-12-04T05:39:59.728835+0000","last_became_peered":"2023-12-04T05:39:59.728835+0000","last_unstale":"2023-12-04T05:40:01.502427+0000","last_undegraded":"2023-12-04T05:40:01.502427+0000","last_fullsized":"2023-12-04T05:40:01.502427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:37:37.490362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501933+0000","last_change":"2023-12-04T05:39:59.724592+0000","last_active":"2023-12-04T05:40:01.501933+0000","last_peered":"2023-12-04T05:40:01.501933+0000","last_clean":"2023-12-04T05:40:01.501933+0000","last_became_active":"2023-12-04T05:39:59.723876+0000","last_became_peered":"2023-12-04T05:39:59.723876+0000","last_unstale":"2023-12-04T05:40:01.501933+0000","last_undegraded":"2023-12-04T05:40:01.501933+0000","last_fullsized":"2023-12-04T05:40:01.501933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:09:26.625728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683703+0000","last_change":"2023-12-04T05:39:59.726540+0000","last_active":"2023-12-04T05:40:00.683703+0000","last_peered":"2023-12-04T05:40:00.683703+0000","last_clean":"2023-12-04T05:40:00.683703+0000","last_became_active":"2023-12-04T05:39:59.725781+0000","last_became_peered":"2023-12-04T05:39:59.725781+0000","last_unstale":"2023-12-04T05:40:00.683703+0000","last_undegraded":"2023-12-04T05:40:00.683703+0000","last_fullsized":"2023-12-04T05:40:00.683703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:59:22.013410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.678751+0000","last_change":"2023-12-04T05:39:59.708376+0000","last_active":"2023-12-04T05:40:00.678751+0000","last_peered":"2023-12-04T05:40:00.678751+0000","last_clean":"2023-12-04T05:40:00.678751+0000","last_became_active":"2023-12-04T05:39:59.707604+0000","last_became_peered":"2023-12-04T05:39:59.707604+0000","last_unstale":"2023-12-04T05:40:00.678751+0000","last_undegraded":"2023-12-04T05:40:00.678751+0000","last_fullsized":"2023-12-04T05:40:00.678751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:00:23.385619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501476+0000","last_change":"2023-12-04T05:39:59.720510+0000","last_active":"2023-12-04T05:40:01.501476+0000","last_peered":"2023-12-04T05:40:01.501476+0000","last_clean":"2023-12-04T05:40:01.501476+0000","last_became_active":"2023-12-04T05:39:59.719704+0000","last_became_peered":"2023-12-04T05:39:59.719704+0000","last_unstale":"2023-12-04T05:40:01.501476+0000","last_undegraded":"2023-12-04T05:40:01.501476+0000","last_fullsized":"2023-12-04T05:40:01.501476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:39:15.714817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681790+0000","last_change":"2023-12-04T05:39:59.711839+0000","last_active":"2023-12-04T05:40:00.681790+0000","last_peered":"2023-12-04T05:40:00.681790+0000","last_clean":"2023-12-04T05:40:00.681790+0000","last_became_active":"2023-12-04T05:39:59.711113+0000","last_became_peered":"2023-12-04T05:39:59.711113+0000","last_unstale":"2023-12-04T05:40:00.681790+0000","last_undegraded":"2023-12-04T05:40:00.681790+0000","last_fullsized":"2023-12-04T05:40:00.681790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:21:17.569525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:10.823687+0000","last_change":"2023-12-04T05:39:58.750554+0000","last_active":"2023-12-04T05:40:10.823687+0000","last_peered":"2023-12-04T05:40:10.823687+0000","last_clean":"2023-12-04T05:40:10.823687+0000","last_became_active":"2023-12-04T05:39:58.750227+0000","last_became_peered":"2023-12-04T05:39:58.750227+0000","last_unstale":"2023-12-04T05:40:10.823687+0000","last_undegraded":"2023-12-04T05:40:10.823687+0000","last_fullsized":"2023-12-04T05:40:10.823687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:13:23.997255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502276+0000","last_change":"2023-12-04T05:39:59.731582+0000","last_active":"2023-12-04T05:40:01.502276+0000","last_peered":"2023-12-04T05:40:01.502276+0000","last_clean":"2023-12-04T05:40:01.502276+0000","last_became_active":"2023-12-04T05:39:59.730822+0000","last_became_peered":"2023-12-04T05:39:59.730822+0000","last_unstale":"2023-12-04T05:40:01.502276+0000","last_undegraded":"2023-12-04T05:40:01.502276+0000","last_fullsized":"2023-12-04T05:40:01.502276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:10:55.996104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.909876+0000","last_change":"2023-12-04T05:39:58.740011+0000","last_active":"2023-12-04T05:40:22.909876+0000","last_peered":"2023-12-04T05:40:22.909876+0000","last_clean":"2023-12-04T05:40:22.909876+0000","last_became_active":"2023-12-04T05:39:58.739395+0000","last_became_peered":"2023-12-04T05:39:58.739395+0000","last_unstale":"2023-12-04T05:40:22.909876+0000","last_undegraded":"2023-12-04T05:40:22.909876+0000","last_fullsized":"2023-12-04T05:40:22.909876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:17:48.107311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:16.865405+0000","last_change":"2023-12-04T05:39:58.730186+0000","last_active":"2023-12-04T05:40:16.865405+0000","last_peered":"2023-12-04T05:40:16.865405+0000","last_clean":"2023-12-04T05:40:16.865405+0000","last_became_active":"2023-12-04T05:39:58.729411+0000","last_became_peered":"2023-12-04T05:39:58.729411+0000","last_unstale":"2023-12-04T05:40:16.865405+0000","last_undegraded":"2023-12-04T05:40:16.865405+0000","last_fullsized":"2023-12-04T05:40:16.865405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:39:05.463010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502530+0000","last_change":"2023-12-04T05:39:59.715596+0000","last_active":"2023-12-04T05:40:01.502530+0000","last_peered":"2023-12-04T05:40:01.502530+0000","last_clean":"2023-12-04T05:40:01.502530+0000","last_became_active":"2023-12-04T05:39:59.714842+0000","last_became_peered":"2023-12-04T05:39:59.714842+0000","last_unstale":"2023-12-04T05:40:01.502530+0000","last_undegraded":"2023-12-04T05:40:01.502530+0000","last_fullsized":"2023-12-04T05:40:01.502530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:51:10.874053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.903050+0000","last_change":"2023-12-04T05:39:58.722113+0000","last_active":"2023-12-04T05:40:22.903050+0000","last_peered":"2023-12-04T05:40:22.903050+0000","last_clean":"2023-12-04T05:40:22.903050+0000","last_became_active":"2023-12-04T05:39:58.721346+0000","last_became_peered":"2023-12-04T05:39:58.721346+0000","last_unstale":"2023-12-04T05:40:22.903050+0000","last_undegraded":"2023-12-04T05:40:22.903050+0000","last_fullsized":"2023-12-04T05:40:22.903050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:30:11.578047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908461+0000","last_change":"2023-12-04T05:39:59.708575+0000","last_active":"2023-12-04T05:40:00.908461+0000","last_peered":"2023-12-04T05:40:00.908461+0000","last_clean":"2023-12-04T05:40:00.908461+0000","last_became_active":"2023-12-04T05:39:59.707753+0000","last_became_peered":"2023-12-04T05:39:59.707753+0000","last_unstale":"2023-12-04T05:40:00.908461+0000","last_undegraded":"2023-12-04T05:40:00.908461+0000","last_fullsized":"2023-12-04T05:40:00.908461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:47:58.349368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:10.822709+0000","last_change":"2023-12-04T05:39:58.734739+0000","last_active":"2023-12-04T05:40:10.822709+0000","last_peered":"2023-12-04T05:40:10.822709+0000","last_clean":"2023-12-04T05:40:10.822709+0000","last_became_active":"2023-12-04T05:39:58.733897+0000","last_became_peered":"2023-12-04T05:39:58.733897+0000","last_unstale":"2023-12-04T05:40:10.822709+0000","last_undegraded":"2023-12-04T05:40:10.822709+0000","last_fullsized":"2023-12-04T05:40:10.822709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:52:15.157364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683392+0000","last_change":"2023-12-04T05:39:59.715796+0000","last_active":"2023-12-04T05:40:00.683392+0000","last_peered":"2023-12-04T05:40:00.683392+0000","last_clean":"2023-12-04T05:40:00.683392+0000","last_became_active":"2023-12-04T05:39:59.714951+0000","last_became_peered":"2023-12-04T05:39:59.714951+0000","last_unstale":"2023-12-04T05:40:00.683392+0000","last_undegraded":"2023-12-04T05:40:00.683392+0000","last_fullsized":"2023-12-04T05:40:00.683392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:31:12.475136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.933245+0000","last_change":"2023-12-04T05:39:58.725387+0000","last_active":"2023-12-04T05:40:28.933245+0000","last_peered":"2023-12-04T05:40:28.933245+0000","last_clean":"2023-12-04T05:40:28.933245+0000","last_became_active":"2023-12-04T05:39:58.724672+0000","last_became_peered":"2023-12-04T05:39:58.724672+0000","last_unstale":"2023-12-04T05:40:28.933245+0000","last_undegraded":"2023-12-04T05:40:28.933245+0000","last_fullsized":"2023-12-04T05:40:28.933245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:19:43.112172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503316+0000","last_change":"2023-12-04T05:39:59.718789+0000","last_active":"2023-12-04T05:40:01.503316+0000","last_peered":"2023-12-04T05:40:01.503316+0000","last_clean":"2023-12-04T05:40:01.503316+0000","last_became_active":"2023-12-04T05:39:59.717927+0000","last_became_peered":"2023-12-04T05:39:59.717927+0000","last_unstale":"2023-12-04T05:40:01.503316+0000","last_undegraded":"2023-12-04T05:40:01.503316+0000","last_fullsized":"2023-12-04T05:40:01.503316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:43:08.771461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:10.820444+0000","last_change":"2023-12-04T05:39:58.731710+0000","last_active":"2023-12-04T05:40:10.820444+0000","last_peered":"2023-12-04T05:40:10.820444+0000","last_clean":"2023-12-04T05:40:10.820444+0000","last_became_active":"2023-12-04T05:39:58.731373+0000","last_became_peered":"2023-12-04T05:39:58.731373+0000","last_unstale":"2023-12-04T05:40:10.820444+0000","last_undegraded":"2023-12-04T05:40:10.820444+0000","last_fullsized":"2023-12-04T05:40:10.820444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:03:37.899195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682594+0000","last_change":"2023-12-04T05:39:59.725444+0000","last_active":"2023-12-04T05:40:00.682594+0000","last_peered":"2023-12-04T05:40:00.682594+0000","last_clean":"2023-12-04T05:40:00.682594+0000","last_became_active":"2023-12-04T05:39:59.724620+0000","last_became_peered":"2023-12-04T05:39:59.724620+0000","last_unstale":"2023-12-04T05:40:00.682594+0000","last_undegraded":"2023-12-04T05:40:00.682594+0000","last_fullsized":"2023-12-04T05:40:00.682594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:11:17.092060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.941936+0000","last_change":"2023-12-04T05:39:58.730135+0000","last_active":"2023-12-04T05:40:28.941936+0000","last_peered":"2023-12-04T05:40:28.941936+0000","last_clean":"2023-12-04T05:40:28.941936+0000","last_became_active":"2023-12-04T05:39:58.729295+0000","last_became_peered":"2023-12-04T05:39:58.729295+0000","last_unstale":"2023-12-04T05:40:28.941936+0000","last_undegraded":"2023-12-04T05:40:28.941936+0000","last_fullsized":"2023-12-04T05:40:28.941936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:24:42.912063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502658+0000","last_change":"2023-12-04T05:39:59.731327+0000","last_active":"2023-12-04T05:40:01.502658+0000","last_peered":"2023-12-04T05:40:01.502658+0000","last_clean":"2023-12-04T05:40:01.502658+0000","last_became_active":"2023-12-04T05:39:59.730427+0000","last_became_peered":"2023-12-04T05:39:59.730427+0000","last_unstale":"2023-12-04T05:40:01.502658+0000","last_undegraded":"2023-12-04T05:40:01.502658+0000","last_fullsized":"2023-12-04T05:40:01.502658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:34:45.211951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'9","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.937241+0000","last_change":"2023-12-04T05:39:58.711690+0000","last_active":"2023-12-04T05:40:28.937241+0000","last_peered":"2023-12-04T05:40:28.937241+0000","last_clean":"2023-12-04T05:40:28.937241+0000","last_became_active":"2023-12-04T05:39:58.711305+0000","last_became_peered":"2023-12-04T05:39:58.711305+0000","last_unstale":"2023-12-04T05:40:28.937241+0000","last_undegraded":"2023-12-04T05:40:28.937241+0000","last_fullsized":"2023-12-04T05:40:28.937241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:33:02.159265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501048+0000","last_change":"2023-12-04T05:39:59.702650+0000","last_active":"2023-12-04T05:40:01.501048+0000","last_peered":"2023-12-04T05:40:01.501048+0000","last_clean":"2023-12-04T05:40:01.501048+0000","last_became_active":"2023-12-04T05:39:59.701827+0000","last_became_peered":"2023-12-04T05:39:59.701827+0000","last_unstale":"2023-12-04T05:40:01.501048+0000","last_undegraded":"2023-12-04T05:40:01.501048+0000","last_fullsized":"2023-12-04T05:40:01.501048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:46:45.006062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.902757+0000","last_change":"2023-12-04T05:39:58.729371+0000","last_active":"2023-12-04T05:40:22.902757+0000","last_peered":"2023-12-04T05:40:22.902757+0000","last_clean":"2023-12-04T05:40:22.902757+0000","last_became_active":"2023-12-04T05:39:58.727781+0000","last_became_peered":"2023-12-04T05:39:58.727781+0000","last_unstale":"2023-12-04T05:40:22.902757+0000","last_undegraded":"2023-12-04T05:40:22.902757+0000","last_fullsized":"2023-12-04T05:40:22.902757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:37:29.707707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502646+0000","last_change":"2023-12-04T05:39:59.742080+0000","last_active":"2023-12-04T05:40:01.502646+0000","last_peered":"2023-12-04T05:40:01.502646+0000","last_clean":"2023-12-04T05:40:01.502646+0000","last_became_active":"2023-12-04T05:39:59.741440+0000","last_became_peered":"2023-12-04T05:39:59.741440+0000","last_unstale":"2023-12-04T05:40:01.502646+0000","last_undegraded":"2023-12-04T05:40:01.502646+0000","last_fullsized":"2023-12-04T05:40:01.502646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:29:21.379583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.908886+0000","last_change":"2023-12-04T05:39:58.737926+0000","last_active":"2023-12-04T05:40:22.908886+0000","last_peered":"2023-12-04T05:40:22.908886+0000","last_clean":"2023-12-04T05:40:22.908886+0000","last_became_active":"2023-12-04T05:39:58.737403+0000","last_became_peered":"2023-12-04T05:39:58.737403+0000","last_unstale":"2023-12-04T05:40:22.908886+0000","last_undegraded":"2023-12-04T05:40:22.908886+0000","last_fullsized":"2023-12-04T05:40:22.908886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:10:59.035282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501005+0000","last_change":"2023-12-04T05:39:59.710310+0000","last_active":"2023-12-04T05:40:01.501005+0000","last_peered":"2023-12-04T05:40:01.501005+0000","last_clean":"2023-12-04T05:40:01.501005+0000","last_became_active":"2023-12-04T05:39:59.709497+0000","last_became_peered":"2023-12-04T05:39:59.709497+0000","last_unstale":"2023-12-04T05:40:01.501005+0000","last_undegraded":"2023-12-04T05:40:01.501005+0000","last_fullsized":"2023-12-04T05:40:01.501005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:43:43.708311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'17","reported_seq":29,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.899860+0000","last_change":"2023-12-04T05:39:58.728046+0000","last_active":"2023-12-04T05:40:22.899860+0000","last_peered":"2023-12-04T05:40:22.899860+0000","last_clean":"2023-12-04T05:40:22.899860+0000","last_became_active":"2023-12-04T05:39:58.726747+0000","last_became_peered":"2023-12-04T05:39:58.726747+0000","last_unstale":"2023-12-04T05:40:22.899860+0000","last_undegraded":"2023-12-04T05:40:22.899860+0000","last_fullsized":"2023-12-04T05:40:22.899860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:45:11.024395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501689+0000","last_change":"2023-12-04T05:39:59.741770+0000","last_active":"2023-12-04T05:40:01.501689+0000","last_peered":"2023-12-04T05:40:01.501689+0000","last_clean":"2023-12-04T05:40:01.501689+0000","last_became_active":"2023-12-04T05:39:59.741091+0000","last_became_peered":"2023-12-04T05:39:59.741091+0000","last_unstale":"2023-12-04T05:40:01.501689+0000","last_undegraded":"2023-12-04T05:40:01.501689+0000","last_fullsized":"2023-12-04T05:40:01.501689+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:59:20.747612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.772742+0000","last_change":"2023-12-04T05:39:58.712970+0000","last_active":"2023-12-04T05:40:04.772742+0000","last_peered":"2023-12-04T05:40:04.772742+0000","last_clean":"2023-12-04T05:40:04.772742+0000","last_became_active":"2023-12-04T05:39:58.711759+0000","last_became_peered":"2023-12-04T05:39:58.711759+0000","last_unstale":"2023-12-04T05:40:04.772742+0000","last_undegraded":"2023-12-04T05:40:04.772742+0000","last_fullsized":"2023-12-04T05:40:04.772742+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:02:00.957563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503407+0000","last_change":"2023-12-04T05:39:59.742244+0000","last_active":"2023-12-04T05:40:01.503407+0000","last_peered":"2023-12-04T05:40:01.503407+0000","last_clean":"2023-12-04T05:40:01.503407+0000","last_became_active":"2023-12-04T05:39:59.741379+0000","last_became_peered":"2023-12-04T05:39:59.741379+0000","last_unstale":"2023-12-04T05:40:01.503407+0000","last_undegraded":"2023-12-04T05:40:01.503407+0000","last_fullsized":"2023-12-04T05:40:01.503407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:07:57.685895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.934917+0000","last_change":"2023-12-04T05:39:58.729723+0000","last_active":"2023-12-04T05:40:28.934917+0000","last_peered":"2023-12-04T05:40:28.934917+0000","last_clean":"2023-12-04T05:40:28.934917+0000","last_became_active":"2023-12-04T05:39:58.729410+0000","last_became_peered":"2023-12-04T05:39:58.729410+0000","last_unstale":"2023-12-04T05:40:28.934917+0000","last_undegraded":"2023-12-04T05:40:28.934917+0000","last_fullsized":"2023-12-04T05:40:28.934917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:59:10.799774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503914+0000","last_change":"2023-12-04T05:39:59.741718+0000","last_active":"2023-12-04T05:40:01.503914+0000","last_peered":"2023-12-04T05:40:01.503914+0000","last_clean":"2023-12-04T05:40:01.503914+0000","last_became_active":"2023-12-04T05:39:59.741101+0000","last_became_peered":"2023-12-04T05:39:59.741101+0000","last_unstale":"2023-12-04T05:40:01.503914+0000","last_undegraded":"2023-12-04T05:40:01.503914+0000","last_fullsized":"2023-12-04T05:40:01.503914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:41:03.969161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.900442+0000","last_change":"2023-12-04T05:39:58.737224+0000","last_active":"2023-12-04T05:40:22.900442+0000","last_peered":"2023-12-04T05:40:22.900442+0000","last_clean":"2023-12-04T05:40:22.900442+0000","last_became_active":"2023-12-04T05:39:58.736753+0000","last_became_peered":"2023-12-04T05:39:58.736753+0000","last_unstale":"2023-12-04T05:40:22.900442+0000","last_undegraded":"2023-12-04T05:40:22.900442+0000","last_fullsized":"2023-12-04T05:40:22.900442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:33:34.027796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683405+0000","last_change":"2023-12-04T05:39:59.726776+0000","last_active":"2023-12-04T05:40:00.683405+0000","last_peered":"2023-12-04T05:40:00.683405+0000","last_clean":"2023-12-04T05:40:00.683405+0000","last_became_active":"2023-12-04T05:39:59.725991+0000","last_became_peered":"2023-12-04T05:39:59.725991+0000","last_unstale":"2023-12-04T05:40:00.683405+0000","last_undegraded":"2023-12-04T05:40:00.683405+0000","last_fullsized":"2023-12-04T05:40:00.683405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:36:02.168898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.905810+0000","last_change":"2023-12-04T05:39:58.732942+0000","last_active":"2023-12-04T05:40:22.905810+0000","last_peered":"2023-12-04T05:40:22.905810+0000","last_clean":"2023-12-04T05:40:22.905810+0000","last_became_active":"2023-12-04T05:39:58.732401+0000","last_became_peered":"2023-12-04T05:39:58.732401+0000","last_unstale":"2023-12-04T05:40:22.905810+0000","last_undegraded":"2023-12-04T05:40:22.905810+0000","last_fullsized":"2023-12-04T05:40:22.905810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:23:13.676061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503215+0000","last_change":"2023-12-04T05:39:59.730768+0000","last_active":"2023-12-04T05:40:01.503215+0000","last_peered":"2023-12-04T05:40:01.503215+0000","last_clean":"2023-12-04T05:40:01.503215+0000","last_became_active":"2023-12-04T05:39:59.730207+0000","last_became_peered":"2023-12-04T05:39:59.730207+0000","last_unstale":"2023-12-04T05:40:01.503215+0000","last_undegraded":"2023-12-04T05:40:01.503215+0000","last_fullsized":"2023-12-04T05:40:01.503215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:25:19.473188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T05:39:59.705581+0000","last_change":"2023-12-04T05:39:49.314208+0000","last_active":"2023-12-04T05:39:59.705581+0000","last_peered":"2023-12-04T05:39:59.705581+0000","last_clean":"2023-12-04T05:39:59.705581+0000","last_became_active":"2023-12-04T05:39:49.312262+0000","last_became_peered":"2023-12-04T05:39:49.312262+0000","last_unstale":"2023-12-04T05:39:59.705581+0000","last_undegraded":"2023-12-04T05:39:59.705581+0000","last_fullsized":"2023-12-04T05:39:59.705581+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:48.242612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:48.242612+0000","last_clean_scrub_stamp":"2023-12-04T05:39:48.242612+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:19:26.511345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.907597+0000","last_change":"2023-12-04T05:39:58.725538+0000","last_active":"2023-12-04T05:40:22.907597+0000","last_peered":"2023-12-04T05:40:22.907597+0000","last_clean":"2023-12-04T05:40:22.907597+0000","last_became_active":"2023-12-04T05:39:58.724839+0000","last_became_peered":"2023-12-04T05:39:58.724839+0000","last_unstale":"2023-12-04T05:40:22.907597+0000","last_undegraded":"2023-12-04T05:40:22.907597+0000","last_fullsized":"2023-12-04T05:40:22.907597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:34:48.521117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501800+0000","last_change":"2023-12-04T05:39:59.719475+0000","last_active":"2023-12-04T05:40:01.501800+0000","last_peered":"2023-12-04T05:40:01.501800+0000","last_clean":"2023-12-04T05:40:01.501800+0000","last_became_active":"2023-12-04T05:39:59.718727+0000","last_became_peered":"2023-12-04T05:39:59.718727+0000","last_unstale":"2023-12-04T05:40:01.501800+0000","last_undegraded":"2023-12-04T05:40:01.501800+0000","last_fullsized":"2023-12-04T05:40:01.501800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:53:35.787086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:16.863882+0000","last_change":"2023-12-04T05:39:58.751135+0000","last_active":"2023-12-04T05:40:16.863882+0000","last_peered":"2023-12-04T05:40:16.863882+0000","last_clean":"2023-12-04T05:40:16.863882+0000","last_became_active":"2023-12-04T05:39:58.750144+0000","last_became_peered":"2023-12-04T05:39:58.750144+0000","last_unstale":"2023-12-04T05:40:16.863882+0000","last_undegraded":"2023-12-04T05:40:16.863882+0000","last_fullsized":"2023-12-04T05:40:16.863882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:31:32.599659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.500789+0000","last_change":"2023-12-04T05:39:59.729585+0000","last_active":"2023-12-04T05:40:01.500789+0000","last_peered":"2023-12-04T05:40:01.500789+0000","last_clean":"2023-12-04T05:40:01.500789+0000","last_became_active":"2023-12-04T05:39:59.728381+0000","last_became_peered":"2023-12-04T05:39:59.728381+0000","last_unstale":"2023-12-04T05:40:01.500789+0000","last_undegraded":"2023-12-04T05:40:01.500789+0000","last_fullsized":"2023-12-04T05:40:01.500789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:35:11.920755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:16.867806+0000","last_change":"2023-12-04T05:39:58.731429+0000","last_active":"2023-12-04T05:40:16.867806+0000","last_peered":"2023-12-04T05:40:16.867806+0000","last_clean":"2023-12-04T05:40:16.867806+0000","last_became_active":"2023-12-04T05:39:58.731049+0000","last_became_peered":"2023-12-04T05:39:58.731049+0000","last_unstale":"2023-12-04T05:40:16.867806+0000","last_undegraded":"2023-12-04T05:40:16.867806+0000","last_fullsized":"2023-12-04T05:40:16.867806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:27:16.863460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908531+0000","last_change":"2023-12-04T05:39:59.701324+0000","last_active":"2023-12-04T05:40:00.908531+0000","last_peered":"2023-12-04T05:40:00.908531+0000","last_clean":"2023-12-04T05:40:00.908531+0000","last_became_active":"2023-12-04T05:39:59.700586+0000","last_became_peered":"2023-12-04T05:39:59.700586+0000","last_unstale":"2023-12-04T05:40:00.908531+0000","last_undegraded":"2023-12-04T05:40:00.908531+0000","last_fullsized":"2023-12-04T05:40:00.908531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:37:17.301688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.937384+0000","last_change":"2023-12-04T05:39:58.729635+0000","last_active":"2023-12-04T05:40:28.937384+0000","last_peered":"2023-12-04T05:40:28.937384+0000","last_clean":"2023-12-04T05:40:28.937384+0000","last_became_active":"2023-12-04T05:39:58.728927+0000","last_became_peered":"2023-12-04T05:39:58.728927+0000","last_unstale":"2023-12-04T05:40:28.937384+0000","last_undegraded":"2023-12-04T05:40:28.937384+0000","last_fullsized":"2023-12-04T05:40:28.937384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:58:46.851065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681790+0000","last_change":"2023-12-04T05:39:59.702840+0000","last_active":"2023-12-04T05:40:00.681790+0000","last_peered":"2023-12-04T05:40:00.681790+0000","last_clean":"2023-12-04T05:40:00.681790+0000","last_became_active":"2023-12-04T05:39:59.701652+0000","last_became_peered":"2023-12-04T05:39:59.701652+0000","last_unstale":"2023-12-04T05:40:00.681790+0000","last_undegraded":"2023-12-04T05:40:00.681790+0000","last_fullsized":"2023-12-04T05:40:00.681790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:44:14.733182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.521524+0000","last_change":"2023-12-04T05:39:58.721834+0000","last_active":"2023-12-04T05:40:01.521524+0000","last_peered":"2023-12-04T05:40:01.521524+0000","last_clean":"2023-12-04T05:40:01.521524+0000","last_became_active":"2023-12-04T05:39:58.721451+0000","last_became_peered":"2023-12-04T05:39:58.721451+0000","last_unstale":"2023-12-04T05:40:01.521524+0000","last_undegraded":"2023-12-04T05:40:01.521524+0000","last_fullsized":"2023-12-04T05:40:01.521524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:15:52.065853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.684129+0000","last_change":"2023-12-04T05:39:59.731053+0000","last_active":"2023-12-04T05:40:00.684129+0000","last_peered":"2023-12-04T05:40:00.684129+0000","last_clean":"2023-12-04T05:40:00.684129+0000","last_became_active":"2023-12-04T05:39:59.729732+0000","last_became_peered":"2023-12-04T05:39:59.729732+0000","last_unstale":"2023-12-04T05:40:00.684129+0000","last_undegraded":"2023-12-04T05:40:00.684129+0000","last_fullsized":"2023-12-04T05:40:00.684129+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:57:38.581311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.903625+0000","last_change":"2023-12-04T05:39:58.751126+0000","last_active":"2023-12-04T05:40:22.903625+0000","last_peered":"2023-12-04T05:40:22.903625+0000","last_clean":"2023-12-04T05:40:22.903625+0000","last_became_active":"2023-12-04T05:39:58.750496+0000","last_became_peered":"2023-12-04T05:39:58.750496+0000","last_unstale":"2023-12-04T05:40:22.903625+0000","last_undegraded":"2023-12-04T05:40:22.903625+0000","last_fullsized":"2023-12-04T05:40:22.903625+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:09:22.333993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908579+0000","last_change":"2023-12-04T05:39:59.700032+0000","last_active":"2023-12-04T05:40:00.908579+0000","last_peered":"2023-12-04T05:40:00.908579+0000","last_clean":"2023-12-04T05:40:00.908579+0000","last_became_active":"2023-12-04T05:39:59.699488+0000","last_became_peered":"2023-12-04T05:39:59.699488+0000","last_unstale":"2023-12-04T05:40:00.908579+0000","last_undegraded":"2023-12-04T05:40:00.908579+0000","last_fullsized":"2023-12-04T05:40:00.908579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:59:57.153486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.906694+0000","last_change":"2023-12-04T05:39:58.723126+0000","last_active":"2023-12-04T05:40:22.906694+0000","last_peered":"2023-12-04T05:40:22.906694+0000","last_clean":"2023-12-04T05:40:22.906694+0000","last_became_active":"2023-12-04T05:39:58.722766+0000","last_became_peered":"2023-12-04T05:39:58.722766+0000","last_unstale":"2023-12-04T05:40:22.906694+0000","last_undegraded":"2023-12-04T05:40:22.906694+0000","last_fullsized":"2023-12-04T05:40:22.906694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:28:47.171345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502918+0000","last_change":"2023-12-04T05:39:59.724653+0000","last_active":"2023-12-04T05:40:01.502918+0000","last_peered":"2023-12-04T05:40:01.502918+0000","last_clean":"2023-12-04T05:40:01.502918+0000","last_became_active":"2023-12-04T05:39:59.723945+0000","last_became_peered":"2023-12-04T05:39:59.723945+0000","last_unstale":"2023-12-04T05:40:01.502918+0000","last_undegraded":"2023-12-04T05:40:01.502918+0000","last_fullsized":"2023-12-04T05:40:01.502918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:13:56.812140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.941971+0000","last_change":"2023-12-04T05:39:58.746966+0000","last_active":"2023-12-04T05:40:28.941971+0000","last_peered":"2023-12-04T05:40:28.941971+0000","last_clean":"2023-12-04T05:40:28.941971+0000","last_became_active":"2023-12-04T05:39:58.746261+0000","last_became_peered":"2023-12-04T05:39:58.746261+0000","last_unstale":"2023-12-04T05:40:28.941971+0000","last_undegraded":"2023-12-04T05:40:28.941971+0000","last_fullsized":"2023-12-04T05:40:28.941971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:57:46.899291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503381+0000","last_change":"2023-12-04T05:39:59.713259+0000","last_active":"2023-12-04T05:40:01.503381+0000","last_peered":"2023-12-04T05:40:01.503381+0000","last_clean":"2023-12-04T05:40:01.503381+0000","last_became_active":"2023-12-04T05:39:59.712523+0000","last_became_peered":"2023-12-04T05:39:59.712523+0000","last_unstale":"2023-12-04T05:40:01.503381+0000","last_undegraded":"2023-12-04T05:40:01.503381+0000","last_fullsized":"2023-12-04T05:40:01.503381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:41:10.001755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:10.821316+0000","last_change":"2023-12-04T05:39:58.750282+0000","last_active":"2023-12-04T05:40:10.821316+0000","last_peered":"2023-12-04T05:40:10.821316+0000","last_clean":"2023-12-04T05:40:10.821316+0000","last_became_active":"2023-12-04T05:39:58.749651+0000","last_became_peered":"2023-12-04T05:39:58.749651+0000","last_unstale":"2023-12-04T05:40:10.821316+0000","last_undegraded":"2023-12-04T05:40:10.821316+0000","last_fullsized":"2023-12-04T05:40:10.821316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:40:55.086649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681687+0000","last_change":"2023-12-04T05:39:59.704667+0000","last_active":"2023-12-04T05:40:00.681687+0000","last_peered":"2023-12-04T05:40:00.681687+0000","last_clean":"2023-12-04T05:40:00.681687+0000","last_became_active":"2023-12-04T05:39:59.703943+0000","last_became_peered":"2023-12-04T05:39:59.703943+0000","last_unstale":"2023-12-04T05:40:00.681687+0000","last_undegraded":"2023-12-04T05:40:00.681687+0000","last_fullsized":"2023-12-04T05:40:00.681687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:31:53.890532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'7","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.936677+0000","last_change":"2023-12-04T05:39:58.745277+0000","last_active":"2023-12-04T05:40:28.936677+0000","last_peered":"2023-12-04T05:40:28.936677+0000","last_clean":"2023-12-04T05:40:28.936677+0000","last_became_active":"2023-12-04T05:39:58.744842+0000","last_became_peered":"2023-12-04T05:39:58.744842+0000","last_unstale":"2023-12-04T05:40:28.936677+0000","last_undegraded":"2023-12-04T05:40:28.936677+0000","last_fullsized":"2023-12-04T05:40:28.936677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:50:21.084855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502964+0000","last_change":"2023-12-04T05:39:59.714217+0000","last_active":"2023-12-04T05:40:01.502964+0000","last_peered":"2023-12-04T05:40:01.502964+0000","last_clean":"2023-12-04T05:40:01.502964+0000","last_became_active":"2023-12-04T05:39:59.713438+0000","last_became_peered":"2023-12-04T05:39:59.713438+0000","last_unstale":"2023-12-04T05:40:01.502964+0000","last_undegraded":"2023-12-04T05:40:01.502964+0000","last_fullsized":"2023-12-04T05:40:01.502964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:37:30.790544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.897924+0000","last_change":"2023-12-04T05:39:58.722048+0000","last_active":"2023-12-04T05:40:22.897924+0000","last_peered":"2023-12-04T05:40:22.897924+0000","last_clean":"2023-12-04T05:40:22.897924+0000","last_became_active":"2023-12-04T05:39:58.721782+0000","last_became_peered":"2023-12-04T05:39:58.721782+0000","last_unstale":"2023-12-04T05:40:22.897924+0000","last_undegraded":"2023-12-04T05:40:22.897924+0000","last_fullsized":"2023-12-04T05:40:22.897924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:38:16.023373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502222+0000","last_change":"2023-12-04T05:39:59.715313+0000","last_active":"2023-12-04T05:40:01.502222+0000","last_peered":"2023-12-04T05:40:01.502222+0000","last_clean":"2023-12-04T05:40:01.502222+0000","last_became_active":"2023-12-04T05:39:59.714430+0000","last_became_peered":"2023-12-04T05:39:59.714430+0000","last_unstale":"2023-12-04T05:40:01.502222+0000","last_undegraded":"2023-12-04T05:40:01.502222+0000","last_fullsized":"2023-12-04T05:40:01.502222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:10:01.533077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:10.813719+0000","last_change":"2023-12-04T05:39:58.750921+0000","last_active":"2023-12-04T05:40:10.813719+0000","last_peered":"2023-12-04T05:40:10.813719+0000","last_clean":"2023-12-04T05:40:10.813719+0000","last_became_active":"2023-12-04T05:39:58.750288+0000","last_became_peered":"2023-12-04T05:39:58.750288+0000","last_unstale":"2023-12-04T05:40:10.813719+0000","last_undegraded":"2023-12-04T05:40:10.813719+0000","last_fullsized":"2023-12-04T05:40:10.813719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:04:26.373696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501279+0000","last_change":"2023-12-04T05:39:59.729260+0000","last_active":"2023-12-04T05:40:01.501279+0000","last_peered":"2023-12-04T05:40:01.501279+0000","last_clean":"2023-12-04T05:40:01.501279+0000","last_became_active":"2023-12-04T05:39:59.728639+0000","last_became_peered":"2023-12-04T05:39:59.728639+0000","last_unstale":"2023-12-04T05:40:01.501279+0000","last_undegraded":"2023-12-04T05:40:01.501279+0000","last_fullsized":"2023-12-04T05:40:01.501279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:21:38.039584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683814+0000","last_change":"2023-12-04T05:39:59.731652+0000","last_active":"2023-12-04T05:40:00.683814+0000","last_peered":"2023-12-04T05:40:00.683814+0000","last_clean":"2023-12-04T05:40:00.683814+0000","last_became_active":"2023-12-04T05:39:59.730714+0000","last_became_peered":"2023-12-04T05:39:59.730714+0000","last_unstale":"2023-12-04T05:40:00.683814+0000","last_undegraded":"2023-12-04T05:40:00.683814+0000","last_fullsized":"2023-12-04T05:40:00.683814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:02:27.013221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.936208+0000","last_change":"2023-12-04T05:39:58.751530+0000","last_active":"2023-12-04T05:40:28.936208+0000","last_peered":"2023-12-04T05:40:28.936208+0000","last_clean":"2023-12-04T05:40:28.936208+0000","last_became_active":"2023-12-04T05:39:58.750857+0000","last_became_peered":"2023-12-04T05:39:58.750857+0000","last_unstale":"2023-12-04T05:40:28.936208+0000","last_undegraded":"2023-12-04T05:40:28.936208+0000","last_fullsized":"2023-12-04T05:40:28.936208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:43:36.369813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683195+0000","last_change":"2023-12-04T05:39:59.730535+0000","last_active":"2023-12-04T05:40:00.683195+0000","last_peered":"2023-12-04T05:40:00.683195+0000","last_clean":"2023-12-04T05:40:00.683195+0000","last_became_active":"2023-12-04T05:39:59.729717+0000","last_became_peered":"2023-12-04T05:39:59.729717+0000","last_unstale":"2023-12-04T05:40:00.683195+0000","last_undegraded":"2023-12-04T05:40:00.683195+0000","last_fullsized":"2023-12-04T05:40:00.683195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:31:46.120414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'12","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.942457+0000","last_change":"2023-12-04T05:39:58.752193+0000","last_active":"2023-12-04T05:40:28.942457+0000","last_peered":"2023-12-04T05:40:28.942457+0000","last_clean":"2023-12-04T05:40:28.942457+0000","last_became_active":"2023-12-04T05:39:58.751625+0000","last_became_peered":"2023-12-04T05:39:58.751625+0000","last_unstale":"2023-12-04T05:40:28.942457+0000","last_undegraded":"2023-12-04T05:40:28.942457+0000","last_fullsized":"2023-12-04T05:40:28.942457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:05:05.734494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501167+0000","last_change":"2023-12-04T05:39:59.720197+0000","last_active":"2023-12-04T05:40:01.501167+0000","last_peered":"2023-12-04T05:40:01.501167+0000","last_clean":"2023-12-04T05:40:01.501167+0000","last_became_active":"2023-12-04T05:39:59.718736+0000","last_became_peered":"2023-12-04T05:39:59.718736+0000","last_unstale":"2023-12-04T05:40:01.501167+0000","last_undegraded":"2023-12-04T05:40:01.501167+0000","last_fullsized":"2023-12-04T05:40:01.501167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:41:13.184198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.779055+0000","last_change":"2023-12-04T05:39:58.726906+0000","last_active":"2023-12-04T05:40:04.779055+0000","last_peered":"2023-12-04T05:40:04.779055+0000","last_clean":"2023-12-04T05:40:04.779055+0000","last_became_active":"2023-12-04T05:39:58.726415+0000","last_became_peered":"2023-12-04T05:39:58.726415+0000","last_unstale":"2023-12-04T05:40:04.779055+0000","last_undegraded":"2023-12-04T05:40:04.779055+0000","last_fullsized":"2023-12-04T05:40:04.779055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:58:53.715291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503085+0000","last_change":"2023-12-04T05:39:59.711861+0000","last_active":"2023-12-04T05:40:01.503085+0000","last_peered":"2023-12-04T05:40:01.503085+0000","last_clean":"2023-12-04T05:40:01.503085+0000","last_became_active":"2023-12-04T05:39:59.710933+0000","last_became_peered":"2023-12-04T05:39:59.710933+0000","last_unstale":"2023-12-04T05:40:01.503085+0000","last_undegraded":"2023-12-04T05:40:01.503085+0000","last_fullsized":"2023-12-04T05:40:01.503085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:40:46.361768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'13","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.937128+0000","last_change":"2023-12-04T05:39:58.740704+0000","last_active":"2023-12-04T05:40:28.937128+0000","last_peered":"2023-12-04T05:40:28.937128+0000","last_clean":"2023-12-04T05:40:28.937128+0000","last_became_active":"2023-12-04T05:39:58.740440+0000","last_became_peered":"2023-12-04T05:39:58.740440+0000","last_unstale":"2023-12-04T05:40:28.937128+0000","last_undegraded":"2023-12-04T05:40:28.937128+0000","last_fullsized":"2023-12-04T05:40:28.937128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:20:18.197917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8034,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":205,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":179,"ondisk_log_size":179,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574862,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":847872,"data_stored":668049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.099}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.202}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]}]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":245760,"data_stored":77983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1539999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1859999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":79513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.871}]}]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":241664,"data_stored":76911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1719999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.246}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.018}]}]},{"osd":3,"up_from":13,"seq":55834574862,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":78017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":77043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574862,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":245760,"data_stored":77113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":839680,"data_stored":666823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-04T05:40:42.076 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-04T05:40:42.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f301c879640 1 Processor -- start 2023-12-04T05:40:42.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f301c879640 1 -- start start 2023-12-04T05:40:42.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f301c879640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30180c4c10 0x7f30180c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:42.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f301c879640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30180c55a0 con 0x7f30180d01c0 2023-12-04T05:40:42.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f301c879640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f301815d710 con 0x7f3018154020 2023-12-04T05:40:42.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f301c879640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30180ca230 con 0x7f30180c4c10 2023-12-04T05:40:42.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f3015d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f30180d01c0 0x7f3018071ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39630/0 (socket says 172.21.15.107:39630) 2023-12-04T05:40:42.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f3015d9b640 1 -- 172.21.15.107:0/272005474 learned_addr learned my addr 172.21.15.107:0/272005474 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:42.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f3016d9d640 1 --2- 172.21.15.107:0/272005474 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30180c4c10 0x7f30180c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:42.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f301559a640 1 -- 172.21.15.107:0/272005474 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620659936 0 0) 0x7f30180c55a0 con 0x7f30180d01c0 2023-12-04T05:40:42.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f301559a640 1 -- 172.21.15.107:0/272005474 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3000003530 con 0x7f30180d01c0 2023-12-04T05:40:42.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f301559a640 1 -- 172.21.15.107:0/272005474 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 373295535 0 0) 0x7f3000003530 con 0x7f30180d01c0 2023-12-04T05:40:42.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f301559a640 1 -- 172.21.15.107:0/272005474 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30180c4c10 msgr2=0x7f30180c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:42.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f301559a640 1 --2- 172.21.15.107:0/272005474 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30180c4c10 0x7f30180c5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f301559a640 1 -- 172.21.15.107:0/272005474 >> v1:172.21.15.195:6789/0 conn(0x7f3018154020 legacy=0x7f301815a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:42.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f301559a640 1 -- 172.21.15.107:0/272005474 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301815d9f0 con 0x7f30180d01c0 2023-12-04T05:40:42.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f301559a640 1 -- 172.21.15.107:0/272005474 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f300c0016a0 con 0x7f30180d01c0 2023-12-04T05:40:42.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f301559a640 1 -- 172.21.15.107:0/272005474 >> v1:172.21.15.107:6789/0 conn(0x7f30180d01c0 legacy=0x7f3018071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:42.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f301559a640 1 --2- 172.21.15.107:0/272005474 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30180c4c10 0x7f30000039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:42.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f301559a640 1 --2- 172.21.15.107:0/272005474 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3000003f30 0x7f30000047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:42.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f301559a640 1 --2- 172.21.15.107:0/272005474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3000004d10 0x7f3000005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:42.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f301559a640 1 -- 172.21.15.107:0/272005474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300000bad0 con 0x7f3000004d10 2023-12-04T05:40:42.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f301559a640 1 -- 172.21.15.107:0/272005474 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300000bc40 con 0x7f3000003f30 2023-12-04T05:40:42.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.226+0000 7f301559a640 1 -- 172.21.15.107:0/272005474 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f300000be00 con 0x7f30180c4c10 2023-12-04T05:40:42.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.230+0000 7f3016d9d640 1 --2- 172.21.15.107:0/272005474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3000004d10 0x7f3000005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:42.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.230+0000 7f3015d9b640 1 --2- 172.21.15.107:0/272005474 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3000003f30 0x7f30000047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:42.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.230+0000 7f3016d9d640 1 -- 172.21.15.107:0/272005474 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30180c4c10 msgr2=0x7f30000039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:42.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.230+0000 7f301659c640 1 --2- 172.21.15.107:0/272005474 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30180c4c10 0x7f30000039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:42.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.230+0000 7f3016d9d640 1 --2- 172.21.15.107:0/272005474 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30180c4c10 0x7f30000039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.230+0000 7f3016d9d640 1 -- 172.21.15.107:0/272005474 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3000003f30 msgr2=0x7f30000047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:42.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.230+0000 7f3016d9d640 1 --2- 172.21.15.107:0/272005474 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3000003f30 0x7f30000047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.230+0000 7f3016d9d640 1 -- 172.21.15.107:0/272005474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3000003530 con 0x7f3000004d10 2023-12-04T05:40:42.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.230+0000 7f3015d9b640 1 --2- 172.21.15.107:0/272005474 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3000003f30 0x7f30000047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:42.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.230+0000 7f301659c640 1 --2- 172.21.15.107:0/272005474 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30180c4c10 0x7f30000039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:42.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.230+0000 7f3016d9d640 1 --2- 172.21.15.107:0/272005474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3000004d10 0x7f3000005190 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3010009fc0 tx=0x7f301001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c09e5170b18523e server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:42.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.230+0000 7f301559a640 1 -- 172.21.15.107:0/272005474 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30100a5c70 con 0x7f3000004d10 2023-12-04T05:40:42.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.230+0000 7f301559a640 1 -- 172.21.15.107:0/272005474 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301000f040 con 0x7f3000004d10 2023-12-04T05:40:42.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.230+0000 7f301559a640 1 -- 172.21.15.107:0/272005474 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30100ad440 con 0x7f3000004d10 2023-12-04T05:40:42.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.230+0000 7f301c879640 1 -- 172.21.15.107:0/272005474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3000004d10 msgr2=0x7f3000005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:42.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.230+0000 7f301c879640 1 --2- 172.21.15.107:0/272005474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3000004d10 0x7f3000005190 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3010009fc0 tx=0x7f301001cc60 comp rx=0 tx=0).stop 2023-12-04T05:40:42.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.230+0000 7f3016d9d640 1 -- 172.21.15.107:0/272005474 reap_dead start 2023-12-04T05:40:42.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.230+0000 7f301c879640 1 -- 172.21.15.107:0/272005474 shutdown_connections 2023-12-04T05:40:42.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.230+0000 7f301c879640 1 -- 172.21.15.107:0/272005474 >> 172.21.15.107:0/272005474 conn(0x7f301806c6c0 msgr2=0x7f30180cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:42.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.230+0000 7f301c879640 1 -- 172.21.15.107:0/272005474 shutdown_connections 2023-12-04T05:40:42.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.230+0000 7f301c879640 1 -- 172.21.15.107:0/272005474 wait complete. 2023-12-04T05:40:42.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f301c879640 1 Processor -- start 2023-12-04T05:40:42.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f301c879640 1 -- start start 2023-12-04T05:40:42.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f301c879640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3000004d10 0x7f30180b2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:42.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f301c879640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30180b3e90 0x7f30180ce880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:42.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f301c879640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30180b30f0 0x7f30180b3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:42.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f301c879640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3018154ed0 con 0x7f3000004d10 2023-12-04T05:40:42.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f301c879640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30180b4c60 con 0x7f30180b3e90 2023-12-04T05:40:42.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f301c879640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30180b2c60 con 0x7f30180b30f0 2023-12-04T05:40:42.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f301659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3000004d10 0x7f30180b2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:42.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f301659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3000004d10 0x7f30180b2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56326/0 (socket says 172.21.15.107:56326) 2023-12-04T05:40:42.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f301659c640 1 -- 172.21.15.107:0/2045390264 learned_addr learned my addr 172.21.15.107:0/2045390264 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:42.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f3016d9d640 1 --2- 172.21.15.107:0/2045390264 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30180b30f0 0x7f30180b3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:42.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f301659c640 1 -- 172.21.15.107:0/2045390264 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30180b30f0 msgr2=0x7f30180b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:42.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f301659c640 1 --2- 172.21.15.107:0/2045390264 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30180b30f0 0x7f30180b3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f301659c640 1 -- 172.21.15.107:0/2045390264 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30180b3e90 msgr2=0x7f30180ce880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:42.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f301659c640 1 --2- 172.21.15.107:0/2045390264 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30180b3e90 0x7f30180ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f301659c640 1 -- 172.21.15.107:0/2045390264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30180b4430 con 0x7f3000004d10 2023-12-04T05:40:42.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f301659c640 1 --2- 172.21.15.107:0/2045390264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3000004d10 0x7f30180b2130 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f300800d800 tx=0x7f300800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:42.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f3006ffd640 1 -- 172.21.15.107:0/2045390264 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3008005270 con 0x7f3000004d10 2023-12-04T05:40:42.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f3006ffd640 1 -- 172.21.15.107:0/2045390264 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30080053d0 con 0x7f3000004d10 2023-12-04T05:40:42.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f301c879640 1 -- 172.21.15.107:0/2045390264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30181efdd0 con 0x7f3000004d10 2023-12-04T05:40:42.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f3006ffd640 1 -- 172.21.15.107:0/2045390264 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300800b870 con 0x7f3000004d10 2023-12-04T05:40:42.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f301c879640 1 -- 172.21.15.107:0/2045390264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30180b4980 con 0x7f3000004d10 2023-12-04T05:40:42.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f301c879640 1 -- 172.21.15.107:0/2045390264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30180b2a30 con 0x7f3000004d10 2023-12-04T05:40:42.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f3006ffd640 1 -- 172.21.15.107:0/2045390264 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f300800b9d0 con 0x7f3000004d10 2023-12-04T05:40:42.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f3006ffd640 1 --2- 172.21.15.107:0/2045390264 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2ff8066900 0x7f2ff8068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:42.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f3006ffd640 1 -- 172.21.15.107:0/2045390264 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3008083fc0 con 0x7f3000004d10 2023-12-04T05:40:42.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.234+0000 7f3015d9b640 1 --2- 172.21.15.107:0/2045390264 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2ff8066900 0x7f2ff8068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:42.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.238+0000 7f3015d9b640 1 --2- 172.21.15.107:0/2045390264 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2ff8066900 0x7f2ff8068dc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f300c000f70 tx=0x7f300c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:42.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.238+0000 7f3006ffd640 1 -- 172.21.15.107:0/2045390264 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f30080b5280 con 0x7f3000004d10 2023-12-04T05:40:42.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.390+0000 7f301c879640 1 -- 172.21.15.107:0/2045390264 --> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f301815e290 con 0x7f2ff8066900 2023-12-04T05:40:42.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.402+0000 7f3006ffd640 1 -- 172.21.15.107:0/2045390264 <== mgr.4104 v2:172.21.15.107:6832/2693048026 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264548 (secure 0 0 0) 0x7f300c00d920 con 0x7f2ff8066900 2023-12-04T05:40:42.408 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:42.411 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-12-04T05:40:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.406+0000 7f301c879640 1 -- 172.21.15.107:0/2045390264 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2ff8066900 msgr2=0x7f2ff8068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.406+0000 7f301c879640 1 --2- 172.21.15.107:0/2045390264 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2ff8066900 0x7f2ff8068dc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f300c000f70 tx=0x7f300c009040 comp rx=0 tx=0).stop 2023-12-04T05:40:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.406+0000 7f301c879640 1 -- 172.21.15.107:0/2045390264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3000004d10 msgr2=0x7f30180b2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.406+0000 7f301c879640 1 --2- 172.21.15.107:0/2045390264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3000004d10 0x7f30180b2130 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f300800d800 tx=0x7f300800dcc0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.406+0000 7f301c879640 1 -- 172.21.15.107:0/2045390264 shutdown_connections 2023-12-04T05:40:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.406+0000 7f301c879640 1 --2- 172.21.15.107:0/2045390264 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2ff8066900 0x7f2ff8068dc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.406+0000 7f301c879640 1 --2- 172.21.15.107:0/2045390264 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f30180b30f0 0x7f30180b3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.406+0000 7f301c879640 1 --2- 172.21.15.107:0/2045390264 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f30180b3e90 0x7f30180ce880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.406+0000 7f301c879640 1 --2- 172.21.15.107:0/2045390264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3000004d10 0x7f30180b2130 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.406+0000 7f301c879640 1 -- 172.21.15.107:0/2045390264 >> 172.21.15.107:0/2045390264 conn(0x7f301806c6c0 msgr2=0x7f30180c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.406+0000 7f301c879640 1 -- 172.21.15.107:0/2045390264 shutdown_connections 2023-12-04T05:40:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.406+0000 7f301c879640 1 -- 172.21.15.107:0/2045390264 wait complete. 2023-12-04T05:40:42.428 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2023-12-04T05:40:41.504546+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598402,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":364,"num_write_kb":2478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":296,"ondisk_log_size":296,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235880,"kb_used_data":3112,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738840,"statfs":{"total":773094113280,"available":772852572160,"internally_reserved":0,"allocated":3186688,"data_stored":1801452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002767"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503577+0000","last_change":"2023-12-04T05:39:59.736570+0000","last_active":"2023-12-04T05:40:01.503577+0000","last_peered":"2023-12-04T05:40:01.503577+0000","last_clean":"2023-12-04T05:40:01.503577+0000","last_became_active":"2023-12-04T05:39:59.736000+0000","last_became_peered":"2023-12-04T05:39:59.736000+0000","last_unstale":"2023-12-04T05:40:01.503577+0000","last_undegraded":"2023-12-04T05:40:01.503577+0000","last_fullsized":"2023-12-04T05:40:01.503577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:34:15.458889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908781+0000","last_change":"2023-12-04T05:39:59.703955+0000","last_active":"2023-12-04T05:40:00.908781+0000","last_peered":"2023-12-04T05:40:00.908781+0000","last_clean":"2023-12-04T05:40:00.908781+0000","last_became_active":"2023-12-04T05:39:59.703032+0000","last_became_peered":"2023-12-04T05:39:59.703032+0000","last_unstale":"2023-12-04T05:40:00.908781+0000","last_undegraded":"2023-12-04T05:40:00.908781+0000","last_fullsized":"2023-12-04T05:40:00.908781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:22:09.010348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501353+0000","last_change":"2023-12-04T05:39:59.705340+0000","last_active":"2023-12-04T05:40:01.501353+0000","last_peered":"2023-12-04T05:40:01.501353+0000","last_clean":"2023-12-04T05:40:01.501353+0000","last_became_active":"2023-12-04T05:39:59.699479+0000","last_became_peered":"2023-12-04T05:39:59.699479+0000","last_unstale":"2023-12-04T05:40:01.501353+0000","last_undegraded":"2023-12-04T05:40:01.501353+0000","last_fullsized":"2023-12-04T05:40:01.501353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:12:00.122168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683148+0000","last_change":"2023-12-04T05:39:59.728698+0000","last_active":"2023-12-04T05:40:00.683148+0000","last_peered":"2023-12-04T05:40:00.683148+0000","last_clean":"2023-12-04T05:40:00.683148+0000","last_became_active":"2023-12-04T05:39:59.727914+0000","last_became_peered":"2023-12-04T05:39:59.727914+0000","last_unstale":"2023-12-04T05:40:00.683148+0000","last_undegraded":"2023-12-04T05:40:00.683148+0000","last_fullsized":"2023-12-04T05:40:00.683148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:39:17.269322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502651+0000","last_change":"2023-12-04T05:39:59.726320+0000","last_active":"2023-12-04T05:40:01.502651+0000","last_peered":"2023-12-04T05:40:01.502651+0000","last_clean":"2023-12-04T05:40:01.502651+0000","last_became_active":"2023-12-04T05:39:59.722627+0000","last_became_peered":"2023-12-04T05:39:59.722627+0000","last_unstale":"2023-12-04T05:40:01.502651+0000","last_undegraded":"2023-12-04T05:40:01.502651+0000","last_fullsized":"2023-12-04T05:40:01.502651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:40:48.940938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682860+0000","last_change":"2023-12-04T05:39:59.718363+0000","last_active":"2023-12-04T05:40:00.682860+0000","last_peered":"2023-12-04T05:40:00.682860+0000","last_clean":"2023-12-04T05:40:00.682860+0000","last_became_active":"2023-12-04T05:39:59.717475+0000","last_became_peered":"2023-12-04T05:39:59.717475+0000","last_unstale":"2023-12-04T05:40:00.682860+0000","last_undegraded":"2023-12-04T05:40:00.682860+0000","last_fullsized":"2023-12-04T05:40:00.682860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:02:56.684095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501353+0000","last_change":"2023-12-04T05:39:59.724035+0000","last_active":"2023-12-04T05:40:01.501353+0000","last_peered":"2023-12-04T05:40:01.501353+0000","last_clean":"2023-12-04T05:40:01.501353+0000","last_became_active":"2023-12-04T05:39:59.723277+0000","last_became_peered":"2023-12-04T05:39:59.723277+0000","last_unstale":"2023-12-04T05:40:01.501353+0000","last_undegraded":"2023-12-04T05:40:01.501353+0000","last_fullsized":"2023-12-04T05:40:01.501353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:34:54.379064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502157+0000","last_change":"2023-12-04T05:39:59.720028+0000","last_active":"2023-12-04T05:40:01.502157+0000","last_peered":"2023-12-04T05:40:01.502157+0000","last_clean":"2023-12-04T05:40:01.502157+0000","last_became_active":"2023-12-04T05:39:59.719236+0000","last_became_peered":"2023-12-04T05:39:59.719236+0000","last_unstale":"2023-12-04T05:40:01.502157+0000","last_undegraded":"2023-12-04T05:40:01.502157+0000","last_fullsized":"2023-12-04T05:40:01.502157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:23:39.909425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503754+0000","last_change":"2023-12-04T05:39:59.732301+0000","last_active":"2023-12-04T05:40:01.503754+0000","last_peered":"2023-12-04T05:40:01.503754+0000","last_clean":"2023-12-04T05:40:01.503754+0000","last_became_active":"2023-12-04T05:39:59.731561+0000","last_became_peered":"2023-12-04T05:39:59.731561+0000","last_unstale":"2023-12-04T05:40:01.503754+0000","last_undegraded":"2023-12-04T05:40:01.503754+0000","last_fullsized":"2023-12-04T05:40:01.503754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:49:40.179867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501688+0000","last_change":"2023-12-04T05:39:59.721633+0000","last_active":"2023-12-04T05:40:01.501688+0000","last_peered":"2023-12-04T05:40:01.501688+0000","last_clean":"2023-12-04T05:40:01.501688+0000","last_became_active":"2023-12-04T05:39:59.720846+0000","last_became_peered":"2023-12-04T05:39:59.720846+0000","last_unstale":"2023-12-04T05:40:01.501688+0000","last_undegraded":"2023-12-04T05:40:01.501688+0000","last_fullsized":"2023-12-04T05:40:01.501688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:44:34.893307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501835+0000","last_change":"2023-12-04T05:39:59.721173+0000","last_active":"2023-12-04T05:40:01.501835+0000","last_peered":"2023-12-04T05:40:01.501835+0000","last_clean":"2023-12-04T05:40:01.501835+0000","last_became_active":"2023-12-04T05:39:59.720371+0000","last_became_peered":"2023-12-04T05:39:59.720371+0000","last_unstale":"2023-12-04T05:40:01.501835+0000","last_undegraded":"2023-12-04T05:40:01.501835+0000","last_fullsized":"2023-12-04T05:40:01.501835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:12:23.683644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682272+0000","last_change":"2023-12-04T05:39:59.710269+0000","last_active":"2023-12-04T05:40:00.682272+0000","last_peered":"2023-12-04T05:40:00.682272+0000","last_clean":"2023-12-04T05:40:00.682272+0000","last_became_active":"2023-12-04T05:39:59.709565+0000","last_became_peered":"2023-12-04T05:39:59.709565+0000","last_unstale":"2023-12-04T05:40:00.682272+0000","last_undegraded":"2023-12-04T05:40:00.682272+0000","last_fullsized":"2023-12-04T05:40:00.682272+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:11:13.222803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501884+0000","last_change":"2023-12-04T05:39:59.710484+0000","last_active":"2023-12-04T05:40:01.501884+0000","last_peered":"2023-12-04T05:40:01.501884+0000","last_clean":"2023-12-04T05:40:01.501884+0000","last_became_active":"2023-12-04T05:39:59.709623+0000","last_became_peered":"2023-12-04T05:39:59.709623+0000","last_unstale":"2023-12-04T05:40:01.501884+0000","last_undegraded":"2023-12-04T05:40:01.501884+0000","last_fullsized":"2023-12-04T05:40:01.501884+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:40:05.852673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.500880+0000","last_change":"2023-12-04T05:39:59.715685+0000","last_active":"2023-12-04T05:40:01.500880+0000","last_peered":"2023-12-04T05:40:01.500880+0000","last_clean":"2023-12-04T05:40:01.500880+0000","last_became_active":"2023-12-04T05:39:59.714890+0000","last_became_peered":"2023-12-04T05:39:59.714890+0000","last_unstale":"2023-12-04T05:40:01.500880+0000","last_undegraded":"2023-12-04T05:40:01.500880+0000","last_fullsized":"2023-12-04T05:40:01.500880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:59:34.243902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908703+0000","last_change":"2023-12-04T05:39:59.688405+0000","last_active":"2023-12-04T05:40:00.908703+0000","last_peered":"2023-12-04T05:40:00.908703+0000","last_clean":"2023-12-04T05:40:00.908703+0000","last_became_active":"2023-12-04T05:39:59.687603+0000","last_became_peered":"2023-12-04T05:39:59.687603+0000","last_unstale":"2023-12-04T05:40:00.908703+0000","last_undegraded":"2023-12-04T05:40:00.908703+0000","last_fullsized":"2023-12-04T05:40:00.908703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:44:51.269794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681203+0000","last_change":"2023-12-04T05:39:59.718543+0000","last_active":"2023-12-04T05:40:00.681203+0000","last_peered":"2023-12-04T05:40:00.681203+0000","last_clean":"2023-12-04T05:40:00.681203+0000","last_became_active":"2023-12-04T05:39:59.717826+0000","last_became_peered":"2023-12-04T05:39:59.717826+0000","last_unstale":"2023-12-04T05:40:00.681203+0000","last_undegraded":"2023-12-04T05:40:00.681203+0000","last_fullsized":"2023-12-04T05:40:00.681203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:31:03.248114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.684087+0000","last_change":"2023-12-04T05:39:59.724278+0000","last_active":"2023-12-04T05:40:00.684087+0000","last_peered":"2023-12-04T05:40:00.684087+0000","last_clean":"2023-12-04T05:40:00.684087+0000","last_became_active":"2023-12-04T05:39:59.723384+0000","last_became_peered":"2023-12-04T05:39:59.723384+0000","last_unstale":"2023-12-04T05:40:00.684087+0000","last_undegraded":"2023-12-04T05:40:00.684087+0000","last_fullsized":"2023-12-04T05:40:00.684087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:07:20.576260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.679022+0000","last_change":"2023-12-04T05:39:59.711802+0000","last_active":"2023-12-04T05:40:00.679022+0000","last_peered":"2023-12-04T05:40:00.679022+0000","last_clean":"2023-12-04T05:40:00.679022+0000","last_became_active":"2023-12-04T05:39:59.711044+0000","last_became_peered":"2023-12-04T05:39:59.711044+0000","last_unstale":"2023-12-04T05:40:00.679022+0000","last_undegraded":"2023-12-04T05:40:00.679022+0000","last_fullsized":"2023-12-04T05:40:00.679022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:42:28.221010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503300+0000","last_change":"2023-12-04T05:39:59.725226+0000","last_active":"2023-12-04T05:40:01.503300+0000","last_peered":"2023-12-04T05:40:01.503300+0000","last_clean":"2023-12-04T05:40:01.503300+0000","last_became_active":"2023-12-04T05:39:59.724334+0000","last_became_peered":"2023-12-04T05:39:59.724334+0000","last_unstale":"2023-12-04T05:40:01.503300+0000","last_undegraded":"2023-12-04T05:40:01.503300+0000","last_fullsized":"2023-12-04T05:40:01.503300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:48:30.439285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501624+0000","last_change":"2023-12-04T05:39:59.729858+0000","last_active":"2023-12-04T05:40:01.501624+0000","last_peered":"2023-12-04T05:40:01.501624+0000","last_clean":"2023-12-04T05:40:01.501624+0000","last_became_active":"2023-12-04T05:39:59.728989+0000","last_became_peered":"2023-12-04T05:39:59.728989+0000","last_unstale":"2023-12-04T05:40:01.501624+0000","last_undegraded":"2023-12-04T05:40:01.501624+0000","last_fullsized":"2023-12-04T05:40:01.501624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:42:01.099743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501616+0000","last_change":"2023-12-04T05:39:59.724022+0000","last_active":"2023-12-04T05:40:01.501616+0000","last_peered":"2023-12-04T05:40:01.501616+0000","last_clean":"2023-12-04T05:40:01.501616+0000","last_became_active":"2023-12-04T05:39:59.723120+0000","last_became_peered":"2023-12-04T05:39:59.723120+0000","last_unstale":"2023-12-04T05:40:01.501616+0000","last_undegraded":"2023-12-04T05:40:01.501616+0000","last_fullsized":"2023-12-04T05:40:01.501616+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:13:50.048757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683322+0000","last_change":"2023-12-04T05:39:59.714806+0000","last_active":"2023-12-04T05:40:00.683322+0000","last_peered":"2023-12-04T05:40:00.683322+0000","last_clean":"2023-12-04T05:40:00.683322+0000","last_became_active":"2023-12-04T05:39:59.713917+0000","last_became_peered":"2023-12-04T05:39:59.713917+0000","last_unstale":"2023-12-04T05:40:00.683322+0000","last_undegraded":"2023-12-04T05:40:00.683322+0000","last_fullsized":"2023-12-04T05:40:00.683322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:18:03.128562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501966+0000","last_change":"2023-12-04T05:39:59.729098+0000","last_active":"2023-12-04T05:40:01.501966+0000","last_peered":"2023-12-04T05:40:01.501966+0000","last_clean":"2023-12-04T05:40:01.501966+0000","last_became_active":"2023-12-04T05:39:59.728449+0000","last_became_peered":"2023-12-04T05:39:59.728449+0000","last_unstale":"2023-12-04T05:40:01.501966+0000","last_undegraded":"2023-12-04T05:40:01.501966+0000","last_fullsized":"2023-12-04T05:40:01.501966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:25:12.098431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682551+0000","last_change":"2023-12-04T05:39:59.708300+0000","last_active":"2023-12-04T05:40:00.682551+0000","last_peered":"2023-12-04T05:40:00.682551+0000","last_clean":"2023-12-04T05:40:00.682551+0000","last_became_active":"2023-12-04T05:39:59.707530+0000","last_became_peered":"2023-12-04T05:39:59.707530+0000","last_unstale":"2023-12-04T05:40:00.682551+0000","last_undegraded":"2023-12-04T05:40:00.682551+0000","last_fullsized":"2023-12-04T05:40:00.682551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:11:40.053312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502562+0000","last_change":"2023-12-04T05:39:59.729406+0000","last_active":"2023-12-04T05:40:01.502562+0000","last_peered":"2023-12-04T05:40:01.502562+0000","last_clean":"2023-12-04T05:40:01.502562+0000","last_became_active":"2023-12-04T05:39:59.728677+0000","last_became_peered":"2023-12-04T05:39:59.728677+0000","last_unstale":"2023-12-04T05:40:01.502562+0000","last_undegraded":"2023-12-04T05:40:01.502562+0000","last_fullsized":"2023-12-04T05:40:01.502562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:03:26.954387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501922+0000","last_change":"2023-12-04T05:39:59.733628+0000","last_active":"2023-12-04T05:40:01.501922+0000","last_peered":"2023-12-04T05:40:01.501922+0000","last_clean":"2023-12-04T05:40:01.501922+0000","last_became_active":"2023-12-04T05:39:59.726137+0000","last_became_peered":"2023-12-04T05:39:59.726137+0000","last_unstale":"2023-12-04T05:40:01.501922+0000","last_undegraded":"2023-12-04T05:40:01.501922+0000","last_fullsized":"2023-12-04T05:40:01.501922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:49:17.630085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908788+0000","last_change":"2023-12-04T05:39:59.727169+0000","last_active":"2023-12-04T05:40:00.908788+0000","last_peered":"2023-12-04T05:40:00.908788+0000","last_clean":"2023-12-04T05:40:00.908788+0000","last_became_active":"2023-12-04T05:39:59.725378+0000","last_became_peered":"2023-12-04T05:39:59.725378+0000","last_unstale":"2023-12-04T05:40:00.908788+0000","last_undegraded":"2023-12-04T05:40:00.908788+0000","last_fullsized":"2023-12-04T05:40:00.908788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:23:21.007397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502427+0000","last_change":"2023-12-04T05:39:59.729516+0000","last_active":"2023-12-04T05:40:01.502427+0000","last_peered":"2023-12-04T05:40:01.502427+0000","last_clean":"2023-12-04T05:40:01.502427+0000","last_became_active":"2023-12-04T05:39:59.728835+0000","last_became_peered":"2023-12-04T05:39:59.728835+0000","last_unstale":"2023-12-04T05:40:01.502427+0000","last_undegraded":"2023-12-04T05:40:01.502427+0000","last_fullsized":"2023-12-04T05:40:01.502427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:37:37.490362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501933+0000","last_change":"2023-12-04T05:39:59.724592+0000","last_active":"2023-12-04T05:40:01.501933+0000","last_peered":"2023-12-04T05:40:01.501933+0000","last_clean":"2023-12-04T05:40:01.501933+0000","last_became_active":"2023-12-04T05:39:59.723876+0000","last_became_peered":"2023-12-04T05:39:59.723876+0000","last_unstale":"2023-12-04T05:40:01.501933+0000","last_undegraded":"2023-12-04T05:40:01.501933+0000","last_fullsized":"2023-12-04T05:40:01.501933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:09:26.625728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683703+0000","last_change":"2023-12-04T05:39:59.726540+0000","last_active":"2023-12-04T05:40:00.683703+0000","last_peered":"2023-12-04T05:40:00.683703+0000","last_clean":"2023-12-04T05:40:00.683703+0000","last_became_active":"2023-12-04T05:39:59.725781+0000","last_became_peered":"2023-12-04T05:39:59.725781+0000","last_unstale":"2023-12-04T05:40:00.683703+0000","last_undegraded":"2023-12-04T05:40:00.683703+0000","last_fullsized":"2023-12-04T05:40:00.683703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:59:22.013410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.678751+0000","last_change":"2023-12-04T05:39:59.708376+0000","last_active":"2023-12-04T05:40:00.678751+0000","last_peered":"2023-12-04T05:40:00.678751+0000","last_clean":"2023-12-04T05:40:00.678751+0000","last_became_active":"2023-12-04T05:39:59.707604+0000","last_became_peered":"2023-12-04T05:39:59.707604+0000","last_unstale":"2023-12-04T05:40:00.678751+0000","last_undegraded":"2023-12-04T05:40:00.678751+0000","last_fullsized":"2023-12-04T05:40:00.678751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:00:23.385619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501476+0000","last_change":"2023-12-04T05:39:59.720510+0000","last_active":"2023-12-04T05:40:01.501476+0000","last_peered":"2023-12-04T05:40:01.501476+0000","last_clean":"2023-12-04T05:40:01.501476+0000","last_became_active":"2023-12-04T05:39:59.719704+0000","last_became_peered":"2023-12-04T05:39:59.719704+0000","last_unstale":"2023-12-04T05:40:01.501476+0000","last_undegraded":"2023-12-04T05:40:01.501476+0000","last_fullsized":"2023-12-04T05:40:01.501476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:39:15.714817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681790+0000","last_change":"2023-12-04T05:39:59.711839+0000","last_active":"2023-12-04T05:40:00.681790+0000","last_peered":"2023-12-04T05:40:00.681790+0000","last_clean":"2023-12-04T05:40:00.681790+0000","last_became_active":"2023-12-04T05:39:59.711113+0000","last_became_peered":"2023-12-04T05:39:59.711113+0000","last_unstale":"2023-12-04T05:40:00.681790+0000","last_undegraded":"2023-12-04T05:40:00.681790+0000","last_fullsized":"2023-12-04T05:40:00.681790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:21:17.569525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:10.823687+0000","last_change":"2023-12-04T05:39:58.750554+0000","last_active":"2023-12-04T05:40:10.823687+0000","last_peered":"2023-12-04T05:40:10.823687+0000","last_clean":"2023-12-04T05:40:10.823687+0000","last_became_active":"2023-12-04T05:39:58.750227+0000","last_became_peered":"2023-12-04T05:39:58.750227+0000","last_unstale":"2023-12-04T05:40:10.823687+0000","last_undegraded":"2023-12-04T05:40:10.823687+0000","last_fullsized":"2023-12-04T05:40:10.823687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:13:23.997255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502276+0000","last_change":"2023-12-04T05:39:59.731582+0000","last_active":"2023-12-04T05:40:01.502276+0000","last_peered":"2023-12-04T05:40:01.502276+0000","last_clean":"2023-12-04T05:40:01.502276+0000","last_became_active":"2023-12-04T05:39:59.730822+0000","last_became_peered":"2023-12-04T05:39:59.730822+0000","last_unstale":"2023-12-04T05:40:01.502276+0000","last_undegraded":"2023-12-04T05:40:01.502276+0000","last_fullsized":"2023-12-04T05:40:01.502276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:10:55.996104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.909876+0000","last_change":"2023-12-04T05:39:58.740011+0000","last_active":"2023-12-04T05:40:22.909876+0000","last_peered":"2023-12-04T05:40:22.909876+0000","last_clean":"2023-12-04T05:40:22.909876+0000","last_became_active":"2023-12-04T05:39:58.739395+0000","last_became_peered":"2023-12-04T05:39:58.739395+0000","last_unstale":"2023-12-04T05:40:22.909876+0000","last_undegraded":"2023-12-04T05:40:22.909876+0000","last_fullsized":"2023-12-04T05:40:22.909876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:17:48.107311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:16.865405+0000","last_change":"2023-12-04T05:39:58.730186+0000","last_active":"2023-12-04T05:40:16.865405+0000","last_peered":"2023-12-04T05:40:16.865405+0000","last_clean":"2023-12-04T05:40:16.865405+0000","last_became_active":"2023-12-04T05:39:58.729411+0000","last_became_peered":"2023-12-04T05:39:58.729411+0000","last_unstale":"2023-12-04T05:40:16.865405+0000","last_undegraded":"2023-12-04T05:40:16.865405+0000","last_fullsized":"2023-12-04T05:40:16.865405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:39:05.463010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502530+0000","last_change":"2023-12-04T05:39:59.715596+0000","last_active":"2023-12-04T05:40:01.502530+0000","last_peered":"2023-12-04T05:40:01.502530+0000","last_clean":"2023-12-04T05:40:01.502530+0000","last_became_active":"2023-12-04T05:39:59.714842+0000","last_became_peered":"2023-12-04T05:39:59.714842+0000","last_unstale":"2023-12-04T05:40:01.502530+0000","last_undegraded":"2023-12-04T05:40:01.502530+0000","last_fullsized":"2023-12-04T05:40:01.502530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:51:10.874053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.903050+0000","last_change":"2023-12-04T05:39:58.722113+0000","last_active":"2023-12-04T05:40:22.903050+0000","last_peered":"2023-12-04T05:40:22.903050+0000","last_clean":"2023-12-04T05:40:22.903050+0000","last_became_active":"2023-12-04T05:39:58.721346+0000","last_became_peered":"2023-12-04T05:39:58.721346+0000","last_unstale":"2023-12-04T05:40:22.903050+0000","last_undegraded":"2023-12-04T05:40:22.903050+0000","last_fullsized":"2023-12-04T05:40:22.903050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:30:11.578047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908461+0000","last_change":"2023-12-04T05:39:59.708575+0000","last_active":"2023-12-04T05:40:00.908461+0000","last_peered":"2023-12-04T05:40:00.908461+0000","last_clean":"2023-12-04T05:40:00.908461+0000","last_became_active":"2023-12-04T05:39:59.707753+0000","last_became_peered":"2023-12-04T05:39:59.707753+0000","last_unstale":"2023-12-04T05:40:00.908461+0000","last_undegraded":"2023-12-04T05:40:00.908461+0000","last_fullsized":"2023-12-04T05:40:00.908461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:47:58.349368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:10.822709+0000","last_change":"2023-12-04T05:39:58.734739+0000","last_active":"2023-12-04T05:40:10.822709+0000","last_peered":"2023-12-04T05:40:10.822709+0000","last_clean":"2023-12-04T05:40:10.822709+0000","last_became_active":"2023-12-04T05:39:58.733897+0000","last_became_peered":"2023-12-04T05:39:58.733897+0000","last_unstale":"2023-12-04T05:40:10.822709+0000","last_undegraded":"2023-12-04T05:40:10.822709+0000","last_fullsized":"2023-12-04T05:40:10.822709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:52:15.157364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683392+0000","last_change":"2023-12-04T05:39:59.715796+0000","last_active":"2023-12-04T05:40:00.683392+0000","last_peered":"2023-12-04T05:40:00.683392+0000","last_clean":"2023-12-04T05:40:00.683392+0000","last_became_active":"2023-12-04T05:39:59.714951+0000","last_became_peered":"2023-12-04T05:39:59.714951+0000","last_unstale":"2023-12-04T05:40:00.683392+0000","last_undegraded":"2023-12-04T05:40:00.683392+0000","last_fullsized":"2023-12-04T05:40:00.683392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:31:12.475136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.933245+0000","last_change":"2023-12-04T05:39:58.725387+0000","last_active":"2023-12-04T05:40:28.933245+0000","last_peered":"2023-12-04T05:40:28.933245+0000","last_clean":"2023-12-04T05:40:28.933245+0000","last_became_active":"2023-12-04T05:39:58.724672+0000","last_became_peered":"2023-12-04T05:39:58.724672+0000","last_unstale":"2023-12-04T05:40:28.933245+0000","last_undegraded":"2023-12-04T05:40:28.933245+0000","last_fullsized":"2023-12-04T05:40:28.933245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:19:43.112172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503316+0000","last_change":"2023-12-04T05:39:59.718789+0000","last_active":"2023-12-04T05:40:01.503316+0000","last_peered":"2023-12-04T05:40:01.503316+0000","last_clean":"2023-12-04T05:40:01.503316+0000","last_became_active":"2023-12-04T05:39:59.717927+0000","last_became_peered":"2023-12-04T05:39:59.717927+0000","last_unstale":"2023-12-04T05:40:01.503316+0000","last_undegraded":"2023-12-04T05:40:01.503316+0000","last_fullsized":"2023-12-04T05:40:01.503316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:43:08.771461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:10.820444+0000","last_change":"2023-12-04T05:39:58.731710+0000","last_active":"2023-12-04T05:40:10.820444+0000","last_peered":"2023-12-04T05:40:10.820444+0000","last_clean":"2023-12-04T05:40:10.820444+0000","last_became_active":"2023-12-04T05:39:58.731373+0000","last_became_peered":"2023-12-04T05:39:58.731373+0000","last_unstale":"2023-12-04T05:40:10.820444+0000","last_undegraded":"2023-12-04T05:40:10.820444+0000","last_fullsized":"2023-12-04T05:40:10.820444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:03:37.899195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682594+0000","last_change":"2023-12-04T05:39:59.725444+0000","last_active":"2023-12-04T05:40:00.682594+0000","last_peered":"2023-12-04T05:40:00.682594+0000","last_clean":"2023-12-04T05:40:00.682594+0000","last_became_active":"2023-12-04T05:39:59.724620+0000","last_became_peered":"2023-12-04T05:39:59.724620+0000","last_unstale":"2023-12-04T05:40:00.682594+0000","last_undegraded":"2023-12-04T05:40:00.682594+0000","last_fullsized":"2023-12-04T05:40:00.682594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:11:17.092060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.941936+0000","last_change":"2023-12-04T05:39:58.730135+0000","last_active":"2023-12-04T05:40:28.941936+0000","last_peered":"2023-12-04T05:40:28.941936+0000","last_clean":"2023-12-04T05:40:28.941936+0000","last_became_active":"2023-12-04T05:39:58.729295+0000","last_became_peered":"2023-12-04T05:39:58.729295+0000","last_unstale":"2023-12-04T05:40:28.941936+0000","last_undegraded":"2023-12-04T05:40:28.941936+0000","last_fullsized":"2023-12-04T05:40:28.941936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:24:42.912063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502658+0000","last_change":"2023-12-04T05:39:59.731327+0000","last_active":"2023-12-04T05:40:01.502658+0000","last_peered":"2023-12-04T05:40:01.502658+0000","last_clean":"2023-12-04T05:40:01.502658+0000","last_became_active":"2023-12-04T05:39:59.730427+0000","last_became_peered":"2023-12-04T05:39:59.730427+0000","last_unstale":"2023-12-04T05:40:01.502658+0000","last_undegraded":"2023-12-04T05:40:01.502658+0000","last_fullsized":"2023-12-04T05:40:01.502658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:34:45.211951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'9","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.937241+0000","last_change":"2023-12-04T05:39:58.711690+0000","last_active":"2023-12-04T05:40:28.937241+0000","last_peered":"2023-12-04T05:40:28.937241+0000","last_clean":"2023-12-04T05:40:28.937241+0000","last_became_active":"2023-12-04T05:39:58.711305+0000","last_became_peered":"2023-12-04T05:39:58.711305+0000","last_unstale":"2023-12-04T05:40:28.937241+0000","last_undegraded":"2023-12-04T05:40:28.937241+0000","last_fullsized":"2023-12-04T05:40:28.937241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:33:02.159265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501048+0000","last_change":"2023-12-04T05:39:59.702650+0000","last_active":"2023-12-04T05:40:01.501048+0000","last_peered":"2023-12-04T05:40:01.501048+0000","last_clean":"2023-12-04T05:40:01.501048+0000","last_became_active":"2023-12-04T05:39:59.701827+0000","last_became_peered":"2023-12-04T05:39:59.701827+0000","last_unstale":"2023-12-04T05:40:01.501048+0000","last_undegraded":"2023-12-04T05:40:01.501048+0000","last_fullsized":"2023-12-04T05:40:01.501048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:46:45.006062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.902757+0000","last_change":"2023-12-04T05:39:58.729371+0000","last_active":"2023-12-04T05:40:22.902757+0000","last_peered":"2023-12-04T05:40:22.902757+0000","last_clean":"2023-12-04T05:40:22.902757+0000","last_became_active":"2023-12-04T05:39:58.727781+0000","last_became_peered":"2023-12-04T05:39:58.727781+0000","last_unstale":"2023-12-04T05:40:22.902757+0000","last_undegraded":"2023-12-04T05:40:22.902757+0000","last_fullsized":"2023-12-04T05:40:22.902757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:37:29.707707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502646+0000","last_change":"2023-12-04T05:39:59.742080+0000","last_active":"2023-12-04T05:40:01.502646+0000","last_peered":"2023-12-04T05:40:01.502646+0000","last_clean":"2023-12-04T05:40:01.502646+0000","last_became_active":"2023-12-04T05:39:59.741440+0000","last_became_peered":"2023-12-04T05:39:59.741440+0000","last_unstale":"2023-12-04T05:40:01.502646+0000","last_undegraded":"2023-12-04T05:40:01.502646+0000","last_fullsized":"2023-12-04T05:40:01.502646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:29:21.379583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.908886+0000","last_change":"2023-12-04T05:39:58.737926+0000","last_active":"2023-12-04T05:40:22.908886+0000","last_peered":"2023-12-04T05:40:22.908886+0000","last_clean":"2023-12-04T05:40:22.908886+0000","last_became_active":"2023-12-04T05:39:58.737403+0000","last_became_peered":"2023-12-04T05:39:58.737403+0000","last_unstale":"2023-12-04T05:40:22.908886+0000","last_undegraded":"2023-12-04T05:40:22.908886+0000","last_fullsized":"2023-12-04T05:40:22.908886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:10:59.035282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501005+0000","last_change":"2023-12-04T05:39:59.710310+0000","last_active":"2023-12-04T05:40:01.501005+0000","last_peered":"2023-12-04T05:40:01.501005+0000","last_clean":"2023-12-04T05:40:01.501005+0000","last_became_active":"2023-12-04T05:39:59.709497+0000","last_became_peered":"2023-12-04T05:39:59.709497+0000","last_unstale":"2023-12-04T05:40:01.501005+0000","last_undegraded":"2023-12-04T05:40:01.501005+0000","last_fullsized":"2023-12-04T05:40:01.501005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:43:43.708311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'17","reported_seq":29,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.899860+0000","last_change":"2023-12-04T05:39:58.728046+0000","last_active":"2023-12-04T05:40:22.899860+0000","last_peered":"2023-12-04T05:40:22.899860+0000","last_clean":"2023-12-04T05:40:22.899860+0000","last_became_active":"2023-12-04T05:39:58.726747+0000","last_became_peered":"2023-12-04T05:39:58.726747+0000","last_unstale":"2023-12-04T05:40:22.899860+0000","last_undegraded":"2023-12-04T05:40:22.899860+0000","last_fullsized":"2023-12-04T05:40:22.899860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:45:11.024395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501689+0000","last_change":"2023-12-04T05:39:59.741770+0000","last_active":"2023-12-04T05:40:01.501689+0000","last_peered":"2023-12-04T05:40:01.501689+0000","last_clean":"2023-12-04T05:40:01.501689+0000","last_became_active":"2023-12-04T05:39:59.741091+0000","last_became_peered":"2023-12-04T05:39:59.741091+0000","last_unstale":"2023-12-04T05:40:01.501689+0000","last_undegraded":"2023-12-04T05:40:01.501689+0000","last_fullsized":"2023-12-04T05:40:01.501689+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:59:20.747612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.772742+0000","last_change":"2023-12-04T05:39:58.712970+0000","last_active":"2023-12-04T05:40:04.772742+0000","last_peered":"2023-12-04T05:40:04.772742+0000","last_clean":"2023-12-04T05:40:04.772742+0000","last_became_active":"2023-12-04T05:39:58.711759+0000","last_became_peered":"2023-12-04T05:39:58.711759+0000","last_unstale":"2023-12-04T05:40:04.772742+0000","last_undegraded":"2023-12-04T05:40:04.772742+0000","last_fullsized":"2023-12-04T05:40:04.772742+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:02:00.957563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503407+0000","last_change":"2023-12-04T05:39:59.742244+0000","last_active":"2023-12-04T05:40:01.503407+0000","last_peered":"2023-12-04T05:40:01.503407+0000","last_clean":"2023-12-04T05:40:01.503407+0000","last_became_active":"2023-12-04T05:39:59.741379+0000","last_became_peered":"2023-12-04T05:39:59.741379+0000","last_unstale":"2023-12-04T05:40:01.503407+0000","last_undegraded":"2023-12-04T05:40:01.503407+0000","last_fullsized":"2023-12-04T05:40:01.503407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:07:57.685895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.934917+0000","last_change":"2023-12-04T05:39:58.729723+0000","last_active":"2023-12-04T05:40:28.934917+0000","last_peered":"2023-12-04T05:40:28.934917+0000","last_clean":"2023-12-04T05:40:28.934917+0000","last_became_active":"2023-12-04T05:39:58.729410+0000","last_became_peered":"2023-12-04T05:39:58.729410+0000","last_unstale":"2023-12-04T05:40:28.934917+0000","last_undegraded":"2023-12-04T05:40:28.934917+0000","last_fullsized":"2023-12-04T05:40:28.934917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:59:10.799774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503914+0000","last_change":"2023-12-04T05:39:59.741718+0000","last_active":"2023-12-04T05:40:01.503914+0000","last_peered":"2023-12-04T05:40:01.503914+0000","last_clean":"2023-12-04T05:40:01.503914+0000","last_became_active":"2023-12-04T05:39:59.741101+0000","last_became_peered":"2023-12-04T05:39:59.741101+0000","last_unstale":"2023-12-04T05:40:01.503914+0000","last_undegraded":"2023-12-04T05:40:01.503914+0000","last_fullsized":"2023-12-04T05:40:01.503914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:41:03.969161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.900442+0000","last_change":"2023-12-04T05:39:58.737224+0000","last_active":"2023-12-04T05:40:22.900442+0000","last_peered":"2023-12-04T05:40:22.900442+0000","last_clean":"2023-12-04T05:40:22.900442+0000","last_became_active":"2023-12-04T05:39:58.736753+0000","last_became_peered":"2023-12-04T05:39:58.736753+0000","last_unstale":"2023-12-04T05:40:22.900442+0000","last_undegraded":"2023-12-04T05:40:22.900442+0000","last_fullsized":"2023-12-04T05:40:22.900442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:33:34.027796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683405+0000","last_change":"2023-12-04T05:39:59.726776+0000","last_active":"2023-12-04T05:40:00.683405+0000","last_peered":"2023-12-04T05:40:00.683405+0000","last_clean":"2023-12-04T05:40:00.683405+0000","last_became_active":"2023-12-04T05:39:59.725991+0000","last_became_peered":"2023-12-04T05:39:59.725991+0000","last_unstale":"2023-12-04T05:40:00.683405+0000","last_undegraded":"2023-12-04T05:40:00.683405+0000","last_fullsized":"2023-12-04T05:40:00.683405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:36:02.168898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.905810+0000","last_change":"2023-12-04T05:39:58.732942+0000","last_active":"2023-12-04T05:40:22.905810+0000","last_peered":"2023-12-04T05:40:22.905810+0000","last_clean":"2023-12-04T05:40:22.905810+0000","last_became_active":"2023-12-04T05:39:58.732401+0000","last_became_peered":"2023-12-04T05:39:58.732401+0000","last_unstale":"2023-12-04T05:40:22.905810+0000","last_undegraded":"2023-12-04T05:40:22.905810+0000","last_fullsized":"2023-12-04T05:40:22.905810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:23:13.676061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503215+0000","last_change":"2023-12-04T05:39:59.730768+0000","last_active":"2023-12-04T05:40:01.503215+0000","last_peered":"2023-12-04T05:40:01.503215+0000","last_clean":"2023-12-04T05:40:01.503215+0000","last_became_active":"2023-12-04T05:39:59.730207+0000","last_became_peered":"2023-12-04T05:39:59.730207+0000","last_unstale":"2023-12-04T05:40:01.503215+0000","last_undegraded":"2023-12-04T05:40:01.503215+0000","last_fullsized":"2023-12-04T05:40:01.503215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:25:19.473188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T05:39:59.705581+0000","last_change":"2023-12-04T05:39:49.314208+0000","last_active":"2023-12-04T05:39:59.705581+0000","last_peered":"2023-12-04T05:39:59.705581+0000","last_clean":"2023-12-04T05:39:59.705581+0000","last_became_active":"2023-12-04T05:39:49.312262+0000","last_became_peered":"2023-12-04T05:39:49.312262+0000","last_unstale":"2023-12-04T05:39:59.705581+0000","last_undegraded":"2023-12-04T05:39:59.705581+0000","last_fullsized":"2023-12-04T05:39:59.705581+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:48.242612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:48.242612+0000","last_clean_scrub_stamp":"2023-12-04T05:39:48.242612+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:19:26.511345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.907597+0000","last_change":"2023-12-04T05:39:58.725538+0000","last_active":"2023-12-04T05:40:22.907597+0000","last_peered":"2023-12-04T05:40:22.907597+0000","last_clean":"2023-12-04T05:40:22.907597+0000","last_became_active":"2023-12-04T05:39:58.724839+0000","last_became_peered":"2023-12-04T05:39:58.724839+0000","last_unstale":"2023-12-04T05:40:22.907597+0000","last_undegraded":"2023-12-04T05:40:22.907597+0000","last_fullsized":"2023-12-04T05:40:22.907597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:34:48.521117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501800+0000","last_change":"2023-12-04T05:39:59.719475+0000","last_active":"2023-12-04T05:40:01.501800+0000","last_peered":"2023-12-04T05:40:01.501800+0000","last_clean":"2023-12-04T05:40:01.501800+0000","last_became_active":"2023-12-04T05:39:59.718727+0000","last_became_peered":"2023-12-04T05:39:59.718727+0000","last_unstale":"2023-12-04T05:40:01.501800+0000","last_undegraded":"2023-12-04T05:40:01.501800+0000","last_fullsized":"2023-12-04T05:40:01.501800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:53:35.787086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:16.863882+0000","last_change":"2023-12-04T05:39:58.751135+0000","last_active":"2023-12-04T05:40:16.863882+0000","last_peered":"2023-12-04T05:40:16.863882+0000","last_clean":"2023-12-04T05:40:16.863882+0000","last_became_active":"2023-12-04T05:39:58.750144+0000","last_became_peered":"2023-12-04T05:39:58.750144+0000","last_unstale":"2023-12-04T05:40:16.863882+0000","last_undegraded":"2023-12-04T05:40:16.863882+0000","last_fullsized":"2023-12-04T05:40:16.863882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:31:32.599659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.500789+0000","last_change":"2023-12-04T05:39:59.729585+0000","last_active":"2023-12-04T05:40:01.500789+0000","last_peered":"2023-12-04T05:40:01.500789+0000","last_clean":"2023-12-04T05:40:01.500789+0000","last_became_active":"2023-12-04T05:39:59.728381+0000","last_became_peered":"2023-12-04T05:39:59.728381+0000","last_unstale":"2023-12-04T05:40:01.500789+0000","last_undegraded":"2023-12-04T05:40:01.500789+0000","last_fullsized":"2023-12-04T05:40:01.500789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:35:11.920755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:16.867806+0000","last_change":"2023-12-04T05:39:58.731429+0000","last_active":"2023-12-04T05:40:16.867806+0000","last_peered":"2023-12-04T05:40:16.867806+0000","last_clean":"2023-12-04T05:40:16.867806+0000","last_became_active":"2023-12-04T05:39:58.731049+0000","last_became_peered":"2023-12-04T05:39:58.731049+0000","last_unstale":"2023-12-04T05:40:16.867806+0000","last_undegraded":"2023-12-04T05:40:16.867806+0000","last_fullsized":"2023-12-04T05:40:16.867806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:27:16.863460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908531+0000","last_change":"2023-12-04T05:39:59.701324+0000","last_active":"2023-12-04T05:40:00.908531+0000","last_peered":"2023-12-04T05:40:00.908531+0000","last_clean":"2023-12-04T05:40:00.908531+0000","last_became_active":"2023-12-04T05:39:59.700586+0000","last_became_peered":"2023-12-04T05:39:59.700586+0000","last_unstale":"2023-12-04T05:40:00.908531+0000","last_undegraded":"2023-12-04T05:40:00.908531+0000","last_fullsized":"2023-12-04T05:40:00.908531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:37:17.301688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.937384+0000","last_change":"2023-12-04T05:39:58.729635+0000","last_active":"2023-12-04T05:40:28.937384+0000","last_peered":"2023-12-04T05:40:28.937384+0000","last_clean":"2023-12-04T05:40:28.937384+0000","last_became_active":"2023-12-04T05:39:58.728927+0000","last_became_peered":"2023-12-04T05:39:58.728927+0000","last_unstale":"2023-12-04T05:40:28.937384+0000","last_undegraded":"2023-12-04T05:40:28.937384+0000","last_fullsized":"2023-12-04T05:40:28.937384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:58:46.851065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681790+0000","last_change":"2023-12-04T05:39:59.702840+0000","last_active":"2023-12-04T05:40:00.681790+0000","last_peered":"2023-12-04T05:40:00.681790+0000","last_clean":"2023-12-04T05:40:00.681790+0000","last_became_active":"2023-12-04T05:39:59.701652+0000","last_became_peered":"2023-12-04T05:39:59.701652+0000","last_unstale":"2023-12-04T05:40:00.681790+0000","last_undegraded":"2023-12-04T05:40:00.681790+0000","last_fullsized":"2023-12-04T05:40:00.681790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:44:14.733182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.521524+0000","last_change":"2023-12-04T05:39:58.721834+0000","last_active":"2023-12-04T05:40:01.521524+0000","last_peered":"2023-12-04T05:40:01.521524+0000","last_clean":"2023-12-04T05:40:01.521524+0000","last_became_active":"2023-12-04T05:39:58.721451+0000","last_became_peered":"2023-12-04T05:39:58.721451+0000","last_unstale":"2023-12-04T05:40:01.521524+0000","last_undegraded":"2023-12-04T05:40:01.521524+0000","last_fullsized":"2023-12-04T05:40:01.521524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:15:52.065853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.684129+0000","last_change":"2023-12-04T05:39:59.731053+0000","last_active":"2023-12-04T05:40:00.684129+0000","last_peered":"2023-12-04T05:40:00.684129+0000","last_clean":"2023-12-04T05:40:00.684129+0000","last_became_active":"2023-12-04T05:39:59.729732+0000","last_became_peered":"2023-12-04T05:39:59.729732+0000","last_unstale":"2023-12-04T05:40:00.684129+0000","last_undegraded":"2023-12-04T05:40:00.684129+0000","last_fullsized":"2023-12-04T05:40:00.684129+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:57:38.581311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.903625+0000","last_change":"2023-12-04T05:39:58.751126+0000","last_active":"2023-12-04T05:40:22.903625+0000","last_peered":"2023-12-04T05:40:22.903625+0000","last_clean":"2023-12-04T05:40:22.903625+0000","last_became_active":"2023-12-04T05:39:58.750496+0000","last_became_peered":"2023-12-04T05:39:58.750496+0000","last_unstale":"2023-12-04T05:40:22.903625+0000","last_undegraded":"2023-12-04T05:40:22.903625+0000","last_fullsized":"2023-12-04T05:40:22.903625+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:09:22.333993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908579+0000","last_change":"2023-12-04T05:39:59.700032+0000","last_active":"2023-12-04T05:40:00.908579+0000","last_peered":"2023-12-04T05:40:00.908579+0000","last_clean":"2023-12-04T05:40:00.908579+0000","last_became_active":"2023-12-04T05:39:59.699488+0000","last_became_peered":"2023-12-04T05:39:59.699488+0000","last_unstale":"2023-12-04T05:40:00.908579+0000","last_undegraded":"2023-12-04T05:40:00.908579+0000","last_fullsized":"2023-12-04T05:40:00.908579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:59:57.153486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.906694+0000","last_change":"2023-12-04T05:39:58.723126+0000","last_active":"2023-12-04T05:40:22.906694+0000","last_peered":"2023-12-04T05:40:22.906694+0000","last_clean":"2023-12-04T05:40:22.906694+0000","last_became_active":"2023-12-04T05:39:58.722766+0000","last_became_peered":"2023-12-04T05:39:58.722766+0000","last_unstale":"2023-12-04T05:40:22.906694+0000","last_undegraded":"2023-12-04T05:40:22.906694+0000","last_fullsized":"2023-12-04T05:40:22.906694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:28:47.171345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502918+0000","last_change":"2023-12-04T05:39:59.724653+0000","last_active":"2023-12-04T05:40:01.502918+0000","last_peered":"2023-12-04T05:40:01.502918+0000","last_clean":"2023-12-04T05:40:01.502918+0000","last_became_active":"2023-12-04T05:39:59.723945+0000","last_became_peered":"2023-12-04T05:39:59.723945+0000","last_unstale":"2023-12-04T05:40:01.502918+0000","last_undegraded":"2023-12-04T05:40:01.502918+0000","last_fullsized":"2023-12-04T05:40:01.502918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:13:56.812140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.941971+0000","last_change":"2023-12-04T05:39:58.746966+0000","last_active":"2023-12-04T05:40:28.941971+0000","last_peered":"2023-12-04T05:40:28.941971+0000","last_clean":"2023-12-04T05:40:28.941971+0000","last_became_active":"2023-12-04T05:39:58.746261+0000","last_became_peered":"2023-12-04T05:39:58.746261+0000","last_unstale":"2023-12-04T05:40:28.941971+0000","last_undegraded":"2023-12-04T05:40:28.941971+0000","last_fullsized":"2023-12-04T05:40:28.941971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:57:46.899291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503381+0000","last_change":"2023-12-04T05:39:59.713259+0000","last_active":"2023-12-04T05:40:01.503381+0000","last_peered":"2023-12-04T05:40:01.503381+0000","last_clean":"2023-12-04T05:40:01.503381+0000","last_became_active":"2023-12-04T05:39:59.712523+0000","last_became_peered":"2023-12-04T05:39:59.712523+0000","last_unstale":"2023-12-04T05:40:01.503381+0000","last_undegraded":"2023-12-04T05:40:01.503381+0000","last_fullsized":"2023-12-04T05:40:01.503381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:41:10.001755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:10.821316+0000","last_change":"2023-12-04T05:39:58.750282+0000","last_active":"2023-12-04T05:40:10.821316+0000","last_peered":"2023-12-04T05:40:10.821316+0000","last_clean":"2023-12-04T05:40:10.821316+0000","last_became_active":"2023-12-04T05:39:58.749651+0000","last_became_peered":"2023-12-04T05:39:58.749651+0000","last_unstale":"2023-12-04T05:40:10.821316+0000","last_undegraded":"2023-12-04T05:40:10.821316+0000","last_fullsized":"2023-12-04T05:40:10.821316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:40:55.086649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681687+0000","last_change":"2023-12-04T05:39:59.704667+0000","last_active":"2023-12-04T05:40:00.681687+0000","last_peered":"2023-12-04T05:40:00.681687+0000","last_clean":"2023-12-04T05:40:00.681687+0000","last_became_active":"2023-12-04T05:39:59.703943+0000","last_became_peered":"2023-12-04T05:39:59.703943+0000","last_unstale":"2023-12-04T05:40:00.681687+0000","last_undegraded":"2023-12-04T05:40:00.681687+0000","last_fullsized":"2023-12-04T05:40:00.681687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:31:53.890532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'7","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.936677+0000","last_change":"2023-12-04T05:39:58.745277+0000","last_active":"2023-12-04T05:40:28.936677+0000","last_peered":"2023-12-04T05:40:28.936677+0000","last_clean":"2023-12-04T05:40:28.936677+0000","last_became_active":"2023-12-04T05:39:58.744842+0000","last_became_peered":"2023-12-04T05:39:58.744842+0000","last_unstale":"2023-12-04T05:40:28.936677+0000","last_undegraded":"2023-12-04T05:40:28.936677+0000","last_fullsized":"2023-12-04T05:40:28.936677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:50:21.084855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502964+0000","last_change":"2023-12-04T05:39:59.714217+0000","last_active":"2023-12-04T05:40:01.502964+0000","last_peered":"2023-12-04T05:40:01.502964+0000","last_clean":"2023-12-04T05:40:01.502964+0000","last_became_active":"2023-12-04T05:39:59.713438+0000","last_became_peered":"2023-12-04T05:39:59.713438+0000","last_unstale":"2023-12-04T05:40:01.502964+0000","last_undegraded":"2023-12-04T05:40:01.502964+0000","last_fullsized":"2023-12-04T05:40:01.502964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:37:30.790544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.897924+0000","last_change":"2023-12-04T05:39:58.722048+0000","last_active":"2023-12-04T05:40:22.897924+0000","last_peered":"2023-12-04T05:40:22.897924+0000","last_clean":"2023-12-04T05:40:22.897924+0000","last_became_active":"2023-12-04T05:39:58.721782+0000","last_became_peered":"2023-12-04T05:39:58.721782+0000","last_unstale":"2023-12-04T05:40:22.897924+0000","last_undegraded":"2023-12-04T05:40:22.897924+0000","last_fullsized":"2023-12-04T05:40:22.897924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:38:16.023373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502222+0000","last_change":"2023-12-04T05:39:59.715313+0000","last_active":"2023-12-04T05:40:01.502222+0000","last_peered":"2023-12-04T05:40:01.502222+0000","last_clean":"2023-12-04T05:40:01.502222+0000","last_became_active":"2023-12-04T05:39:59.714430+0000","last_became_peered":"2023-12-04T05:39:59.714430+0000","last_unstale":"2023-12-04T05:40:01.502222+0000","last_undegraded":"2023-12-04T05:40:01.502222+0000","last_fullsized":"2023-12-04T05:40:01.502222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:10:01.533077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:10.813719+0000","last_change":"2023-12-04T05:39:58.750921+0000","last_active":"2023-12-04T05:40:10.813719+0000","last_peered":"2023-12-04T05:40:10.813719+0000","last_clean":"2023-12-04T05:40:10.813719+0000","last_became_active":"2023-12-04T05:39:58.750288+0000","last_became_peered":"2023-12-04T05:39:58.750288+0000","last_unstale":"2023-12-04T05:40:10.813719+0000","last_undegraded":"2023-12-04T05:40:10.813719+0000","last_fullsized":"2023-12-04T05:40:10.813719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:04:26.373696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501279+0000","last_change":"2023-12-04T05:39:59.729260+0000","last_active":"2023-12-04T05:40:01.501279+0000","last_peered":"2023-12-04T05:40:01.501279+0000","last_clean":"2023-12-04T05:40:01.501279+0000","last_became_active":"2023-12-04T05:39:59.728639+0000","last_became_peered":"2023-12-04T05:39:59.728639+0000","last_unstale":"2023-12-04T05:40:01.501279+0000","last_undegraded":"2023-12-04T05:40:01.501279+0000","last_fullsized":"2023-12-04T05:40:01.501279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:21:38.039584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683814+0000","last_change":"2023-12-04T05:39:59.731652+0000","last_active":"2023-12-04T05:40:00.683814+0000","last_peered":"2023-12-04T05:40:00.683814+0000","last_clean":"2023-12-04T05:40:00.683814+0000","last_became_active":"2023-12-04T05:39:59.730714+0000","last_became_peered":"2023-12-04T05:39:59.730714+0000","last_unstale":"2023-12-04T05:40:00.683814+0000","last_undegraded":"2023-12-04T05:40:00.683814+0000","last_fullsized":"2023-12-04T05:40:00.683814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:02:27.013221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.936208+0000","last_change":"2023-12-04T05:39:58.751530+0000","last_active":"2023-12-04T05:40:28.936208+0000","last_peered":"2023-12-04T05:40:28.936208+0000","last_clean":"2023-12-04T05:40:28.936208+0000","last_became_active":"2023-12-04T05:39:58.750857+0000","last_became_peered":"2023-12-04T05:39:58.750857+0000","last_unstale":"2023-12-04T05:40:28.936208+0000","last_undegraded":"2023-12-04T05:40:28.936208+0000","last_fullsized":"2023-12-04T05:40:28.936208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:43:36.369813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683195+0000","last_change":"2023-12-04T05:39:59.730535+0000","last_active":"2023-12-04T05:40:00.683195+0000","last_peered":"2023-12-04T05:40:00.683195+0000","last_clean":"2023-12-04T05:40:00.683195+0000","last_became_active":"2023-12-04T05:39:59.729717+0000","last_became_peered":"2023-12-04T05:39:59.729717+0000","last_unstale":"2023-12-04T05:40:00.683195+0000","last_undegraded":"2023-12-04T05:40:00.683195+0000","last_fullsized":"2023-12-04T05:40:00.683195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:31:46.120414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'12","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.942457+0000","last_change":"2023-12-04T05:39:58.752193+0000","last_active":"2023-12-04T05:40:28.942457+0000","last_peered":"2023-12-04T05:40:28.942457+0000","last_clean":"2023-12-04T05:40:28.942457+0000","last_became_active":"2023-12-04T05:39:58.751625+0000","last_became_peered":"2023-12-04T05:39:58.751625+0000","last_unstale":"2023-12-04T05:40:28.942457+0000","last_undegraded":"2023-12-04T05:40:28.942457+0000","last_fullsized":"2023-12-04T05:40:28.942457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:05:05.734494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501167+0000","last_change":"2023-12-04T05:39:59.720197+0000","last_active":"2023-12-04T05:40:01.501167+0000","last_peered":"2023-12-04T05:40:01.501167+0000","last_clean":"2023-12-04T05:40:01.501167+0000","last_became_active":"2023-12-04T05:39:59.718736+0000","last_became_peered":"2023-12-04T05:39:59.718736+0000","last_unstale":"2023-12-04T05:40:01.501167+0000","last_undegraded":"2023-12-04T05:40:01.501167+0000","last_fullsized":"2023-12-04T05:40:01.501167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:41:13.184198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.779055+0000","last_change":"2023-12-04T05:39:58.726906+0000","last_active":"2023-12-04T05:40:04.779055+0000","last_peered":"2023-12-04T05:40:04.779055+0000","last_clean":"2023-12-04T05:40:04.779055+0000","last_became_active":"2023-12-04T05:39:58.726415+0000","last_became_peered":"2023-12-04T05:39:58.726415+0000","last_unstale":"2023-12-04T05:40:04.779055+0000","last_undegraded":"2023-12-04T05:40:04.779055+0000","last_fullsized":"2023-12-04T05:40:04.779055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:58:53.715291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503085+0000","last_change":"2023-12-04T05:39:59.711861+0000","last_active":"2023-12-04T05:40:01.503085+0000","last_peered":"2023-12-04T05:40:01.503085+0000","last_clean":"2023-12-04T05:40:01.503085+0000","last_became_active":"2023-12-04T05:39:59.710933+0000","last_became_peered":"2023-12-04T05:39:59.710933+0000","last_unstale":"2023-12-04T05:40:01.503085+0000","last_undegraded":"2023-12-04T05:40:01.503085+0000","last_fullsized":"2023-12-04T05:40:01.503085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:40:46.361768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'13","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.937128+0000","last_change":"2023-12-04T05:39:58.740704+0000","last_active":"2023-12-04T05:40:28.937128+0000","last_peered":"2023-12-04T05:40:28.937128+0000","last_clean":"2023-12-04T05:40:28.937128+0000","last_became_active":"2023-12-04T05:39:58.740440+0000","last_became_peered":"2023-12-04T05:39:58.740440+0000","last_unstale":"2023-12-04T05:40:28.937128+0000","last_undegraded":"2023-12-04T05:40:28.937128+0000","last_fullsized":"2023-12-04T05:40:28.937128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:20:18.197917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8034,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":205,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":179,"ondisk_log_size":179,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574862,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":847872,"data_stored":668049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.099}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.202}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]}]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":245760,"data_stored":77983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1539999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1859999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":79513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.871}]}]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":241664,"data_stored":76911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1719999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.246}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.018}]}]},{"osd":3,"up_from":13,"seq":55834574862,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":78017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":77043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574862,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":245760,"data_stored":77113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":839680,"data_stored":666823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-04T05:40:42.432 INFO:tasks.mds_thrash.ceph_manager:clean! 2023-12-04T05:40:42.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-04T05:40:42.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.590+0000 7f43b48b1640 1 Processor -- start 2023-12-04T05:40:42.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.590+0000 7f43b48b1640 1 -- start start 2023-12-04T05:40:42.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.590+0000 7f43b48b1640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f43b0151e40 0x7f43b00c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:42.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.590+0000 7f43b48b1640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43b00c4f50 0x7f43b00cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:42.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.590+0000 7f43b48b1640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b0072730 con 0x7f43b00c97e0 2023-12-04T05:40:42.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.590+0000 7f43b48b1640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b00c5450 con 0x7f43b0151e40 2023-12-04T05:40:42.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.590+0000 7f43b48b1640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b0158a10 con 0x7f43b00c4f50 2023-12-04T05:40:42.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.590+0000 7f43ae59c640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f43b0151e40 0x7f43b00c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:42.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.590+0000 7f43add9b640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43b00c4f50 0x7f43b00cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:42.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.590+0000 7f43add9b640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43b00c4f50 0x7f43b00cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.107:60764/0 (socket says 172.21.15.107:60764) 2023-12-04T05:40:42.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.590+0000 7f43aed9d640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f43b00c97e0 0x7f43b015b230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:50624/0 (socket says 172.21.15.107:50624) 2023-12-04T05:40:42.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.590+0000 7f43ae59c640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f43b0151e40 0x7f43b00c4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.107:53126/0 (socket says 172.21.15.107:53126) 2023-12-04T05:40:42.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.590+0000 7f43add9b640 1 -- 172.21.15.107:0/1519009529 learned_addr learned my addr 172.21.15.107:0/1519009529 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:42.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.590+0000 7f43ae59c640 1 -- 172.21.15.107:0/1519009529 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43b00c4f50 msgr2=0x7f43b00cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:42.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.590+0000 7f43ae59c640 1 --2- 172.21.15.107:0/1519009529 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43b00c4f50 0x7f43b00cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.590+0000 7f43ae59c640 1 -- 172.21.15.107:0/1519009529 >> v1:172.21.15.195:6789/0 conn(0x7f43b00c97e0 legacy=0x7f43b015b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:42.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.590+0000 7f43ae59c640 1 -- 172.21.15.107:0/1519009529 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b0157d90 con 0x7f43b0151e40 2023-12-04T05:40:42.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.590+0000 7f43add9b640 1 --2- 172.21.15.107:0/1519009529 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43b00c4f50 0x7f43b00cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T05:40:42.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.594+0000 7f43ae59c640 1 --2- 172.21.15.107:0/1519009529 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f43b0151e40 0x7f43b00c4a10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f43a00097c0 tx=0x7f43a001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f703428111fd0bcc server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:42.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.594+0000 7f43ad59a640 1 -- 172.21.15.107:0/1519009529 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a00ab600 con 0x7f43b0151e40 2023-12-04T05:40:42.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.594+0000 7f43ad59a640 1 -- 172.21.15.107:0/1519009529 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a00a7050 con 0x7f43b0151e40 2023-12-04T05:40:42.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.594+0000 7f43ad59a640 1 -- 172.21.15.107:0/1519009529 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a00ab980 con 0x7f43b0151e40 2023-12-04T05:40:42.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.594+0000 7f43b48b1640 1 -- 172.21.15.107:0/1519009529 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f43b0151e40 msgr2=0x7f43b00c4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:42.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.594+0000 7f43b48b1640 1 --2- 172.21.15.107:0/1519009529 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f43b0151e40 0x7f43b00c4a10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f43a00097c0 tx=0x7f43a001c480 comp rx=0 tx=0).stop 2023-12-04T05:40:42.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.594+0000 7f43b48b1640 1 -- 172.21.15.107:0/1519009529 shutdown_connections 2023-12-04T05:40:42.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.594+0000 7f43b48b1640 1 --2- 172.21.15.107:0/1519009529 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43b00c4f50 0x7f43b00cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.594+0000 7f43b48b1640 1 --2- 172.21.15.107:0/1519009529 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f43b0151e40 0x7f43b00c4a10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.594+0000 7f43b48b1640 1 -- 172.21.15.107:0/1519009529 >> 172.21.15.107:0/1519009529 conn(0x7f43b00663a0 msgr2=0x7f43b00667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:42.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.594+0000 7f43b48b1640 1 -- 172.21.15.107:0/1519009529 shutdown_connections 2023-12-04T05:40:42.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.594+0000 7f43b48b1640 1 -- 172.21.15.107:0/1519009529 wait complete. 2023-12-04T05:40:42.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.594+0000 7f43b48b1640 1 Processor -- start 2023-12-04T05:40:42.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.594+0000 7f43b48b1640 1 -- start start 2023-12-04T05:40:42.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43b48b1640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f43b00c4f50 0x7f43b013a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:42.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43b48b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43b00c97e0 0x7f43b013d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:42.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43b48b1640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43b013c060 0x7f43b013c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:42.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43b48b1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b00cbf80 con 0x7f43b00c97e0 2023-12-04T05:40:42.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43b48b1640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b0146d80 con 0x7f43b00c4f50 2023-12-04T05:40:42.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43b48b1640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b00ccbb0 con 0x7f43b013c060 2023-12-04T05:40:42.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43add9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43b00c97e0 0x7f43b013d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:42.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43add9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43b00c97e0 0x7f43b013d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56336/0 (socket says 172.21.15.107:56336) 2023-12-04T05:40:42.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43add9b640 1 -- 172.21.15.107:0/2289561232 learned_addr learned my addr 172.21.15.107:0/2289561232 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:42.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43aed9d640 1 --2- 172.21.15.107:0/2289561232 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43b013c060 0x7f43b013c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:42.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43ae59c640 1 --2- 172.21.15.107:0/2289561232 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f43b00c4f50 0x7f43b013a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:42.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43add9b640 1 -- 172.21.15.107:0/2289561232 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43b013c060 msgr2=0x7f43b013c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:42.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43add9b640 1 --2- 172.21.15.107:0/2289561232 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43b013c060 0x7f43b013c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43add9b640 1 -- 172.21.15.107:0/2289561232 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f43b00c4f50 msgr2=0x7f43b013a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:42.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43add9b640 1 --2- 172.21.15.107:0/2289561232 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f43b00c4f50 0x7f43b013a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43add9b640 1 -- 172.21.15.107:0/2289561232 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43980094d0 con 0x7f43b00c97e0 2023-12-04T05:40:42.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43ae59c640 1 --2- 172.21.15.107:0/2289561232 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f43b00c4f50 0x7f43b013a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:42.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43add9b640 1 --2- 172.21.15.107:0/2289561232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43b00c97e0 0x7f43b013d7a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f439800e930 tx=0x7f439800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:42.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43a6ffd640 1 -- 172.21.15.107:0/2289561232 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439800ca00 con 0x7f43b00c97e0 2023-12-04T05:40:42.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43a6ffd640 1 -- 172.21.15.107:0/2289561232 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f439800cb60 con 0x7f43b00c97e0 2023-12-04T05:40:42.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43a6ffd640 1 -- 172.21.15.107:0/2289561232 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4398013470 con 0x7f43b00c97e0 2023-12-04T05:40:42.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43b48b1640 1 -- 172.21.15.107:0/2289561232 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a0009420 con 0x7f43b00c97e0 2023-12-04T05:40:42.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43b48b1640 1 -- 172.21.15.107:0/2289561232 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b00cb6e0 con 0x7f43b00c97e0 2023-12-04T05:40:42.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43b48b1640 1 -- 172.21.15.107:0/2289561232 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b0146f30 con 0x7f43b00c97e0 2023-12-04T05:40:42.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43a6ffd640 1 -- 172.21.15.107:0/2289561232 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f43980135d0 con 0x7f43b00c97e0 2023-12-04T05:40:42.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43a6ffd640 1 --2- 172.21.15.107:0/2289561232 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f43900668d0 0x7f4390068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:42.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43a6ffd640 1 -- 172.21.15.107:0/2289561232 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4398084670 con 0x7f43b00c97e0 2023-12-04T05:40:42.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43ae59c640 1 --2- 172.21.15.107:0/2289561232 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f43900668d0 0x7f4390068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:42.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.598+0000 7f43ae59c640 1 --2- 172.21.15.107:0/2289561232 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f43900668d0 0x7f4390068d90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f43a00ad000 tx=0x7f43a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:42.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.602+0000 7f43a6ffd640 1 -- 172.21.15.107:0/2289561232 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f43b0146f30 con 0x7f43b00c97e0 2023-12-04T05:40:42.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.758+0000 7f43b48b1640 1 -- 172.21.15.107:0/2289561232 --> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f43b006ee50 con 0x7f43900668d0 2023-12-04T05:40:42.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.766+0000 7f43a6ffd640 1 -- 172.21.15.107:0/2289561232 <== mgr.4104 v2:172.21.15.107:6832/2693048026 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264548 (secure 0 0 0) 0x7f43b006ee50 con 0x7f43900668d0 2023-12-04T05:40:42.773 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:42.776 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-12-04T05:40:42.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.770+0000 7f43b48b1640 1 -- 172.21.15.107:0/2289561232 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f43900668d0 msgr2=0x7f4390068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:42.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.770+0000 7f43b48b1640 1 --2- 172.21.15.107:0/2289561232 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f43900668d0 0x7f4390068d90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f43a00ad000 tx=0x7f43a0008040 comp rx=0 tx=0).stop 2023-12-04T05:40:42.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.770+0000 7f43b48b1640 1 -- 172.21.15.107:0/2289561232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43b00c97e0 msgr2=0x7f43b013d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:42.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.770+0000 7f43b48b1640 1 --2- 172.21.15.107:0/2289561232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43b00c97e0 0x7f43b013d7a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f439800e930 tx=0x7f439800edf0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.770+0000 7f43b48b1640 1 -- 172.21.15.107:0/2289561232 shutdown_connections 2023-12-04T05:40:42.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.770+0000 7f43b48b1640 1 --2- 172.21.15.107:0/2289561232 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f43900668d0 0x7f4390068d90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.770+0000 7f43b48b1640 1 --2- 172.21.15.107:0/2289561232 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f43b013c060 0x7f43b013c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.770+0000 7f43b48b1640 1 --2- 172.21.15.107:0/2289561232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43b00c97e0 0x7f43b013d7a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.770+0000 7f43b48b1640 1 --2- 172.21.15.107:0/2289561232 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f43b00c4f50 0x7f43b013a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.774+0000 7f43b48b1640 1 -- 172.21.15.107:0/2289561232 >> 172.21.15.107:0/2289561232 conn(0x7f43b00663a0 msgr2=0x7f43b0066780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:42.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.774+0000 7f43b48b1640 1 -- 172.21.15.107:0/2289561232 shutdown_connections 2023-12-04T05:40:42.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.774+0000 7f43b48b1640 1 -- 172.21.15.107:0/2289561232 wait complete. 2023-12-04T05:40:42.793 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2023-12-04T05:40:41.504546+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598402,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":364,"num_write_kb":2478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":296,"ondisk_log_size":296,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235880,"kb_used_data":3112,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738840,"statfs":{"total":773094113280,"available":772852572160,"internally_reserved":0,"allocated":3186688,"data_stored":1801452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002767"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503577+0000","last_change":"2023-12-04T05:39:59.736570+0000","last_active":"2023-12-04T05:40:01.503577+0000","last_peered":"2023-12-04T05:40:01.503577+0000","last_clean":"2023-12-04T05:40:01.503577+0000","last_became_active":"2023-12-04T05:39:59.736000+0000","last_became_peered":"2023-12-04T05:39:59.736000+0000","last_unstale":"2023-12-04T05:40:01.503577+0000","last_undegraded":"2023-12-04T05:40:01.503577+0000","last_fullsized":"2023-12-04T05:40:01.503577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:34:15.458889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908781+0000","last_change":"2023-12-04T05:39:59.703955+0000","last_active":"2023-12-04T05:40:00.908781+0000","last_peered":"2023-12-04T05:40:00.908781+0000","last_clean":"2023-12-04T05:40:00.908781+0000","last_became_active":"2023-12-04T05:39:59.703032+0000","last_became_peered":"2023-12-04T05:39:59.703032+0000","last_unstale":"2023-12-04T05:40:00.908781+0000","last_undegraded":"2023-12-04T05:40:00.908781+0000","last_fullsized":"2023-12-04T05:40:00.908781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:22:09.010348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501353+0000","last_change":"2023-12-04T05:39:59.705340+0000","last_active":"2023-12-04T05:40:01.501353+0000","last_peered":"2023-12-04T05:40:01.501353+0000","last_clean":"2023-12-04T05:40:01.501353+0000","last_became_active":"2023-12-04T05:39:59.699479+0000","last_became_peered":"2023-12-04T05:39:59.699479+0000","last_unstale":"2023-12-04T05:40:01.501353+0000","last_undegraded":"2023-12-04T05:40:01.501353+0000","last_fullsized":"2023-12-04T05:40:01.501353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:12:00.122168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683148+0000","last_change":"2023-12-04T05:39:59.728698+0000","last_active":"2023-12-04T05:40:00.683148+0000","last_peered":"2023-12-04T05:40:00.683148+0000","last_clean":"2023-12-04T05:40:00.683148+0000","last_became_active":"2023-12-04T05:39:59.727914+0000","last_became_peered":"2023-12-04T05:39:59.727914+0000","last_unstale":"2023-12-04T05:40:00.683148+0000","last_undegraded":"2023-12-04T05:40:00.683148+0000","last_fullsized":"2023-12-04T05:40:00.683148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:39:17.269322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502651+0000","last_change":"2023-12-04T05:39:59.726320+0000","last_active":"2023-12-04T05:40:01.502651+0000","last_peered":"2023-12-04T05:40:01.502651+0000","last_clean":"2023-12-04T05:40:01.502651+0000","last_became_active":"2023-12-04T05:39:59.722627+0000","last_became_peered":"2023-12-04T05:39:59.722627+0000","last_unstale":"2023-12-04T05:40:01.502651+0000","last_undegraded":"2023-12-04T05:40:01.502651+0000","last_fullsized":"2023-12-04T05:40:01.502651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:40:48.940938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682860+0000","last_change":"2023-12-04T05:39:59.718363+0000","last_active":"2023-12-04T05:40:00.682860+0000","last_peered":"2023-12-04T05:40:00.682860+0000","last_clean":"2023-12-04T05:40:00.682860+0000","last_became_active":"2023-12-04T05:39:59.717475+0000","last_became_peered":"2023-12-04T05:39:59.717475+0000","last_unstale":"2023-12-04T05:40:00.682860+0000","last_undegraded":"2023-12-04T05:40:00.682860+0000","last_fullsized":"2023-12-04T05:40:00.682860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:02:56.684095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501353+0000","last_change":"2023-12-04T05:39:59.724035+0000","last_active":"2023-12-04T05:40:01.501353+0000","last_peered":"2023-12-04T05:40:01.501353+0000","last_clean":"2023-12-04T05:40:01.501353+0000","last_became_active":"2023-12-04T05:39:59.723277+0000","last_became_peered":"2023-12-04T05:39:59.723277+0000","last_unstale":"2023-12-04T05:40:01.501353+0000","last_undegraded":"2023-12-04T05:40:01.501353+0000","last_fullsized":"2023-12-04T05:40:01.501353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:34:54.379064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502157+0000","last_change":"2023-12-04T05:39:59.720028+0000","last_active":"2023-12-04T05:40:01.502157+0000","last_peered":"2023-12-04T05:40:01.502157+0000","last_clean":"2023-12-04T05:40:01.502157+0000","last_became_active":"2023-12-04T05:39:59.719236+0000","last_became_peered":"2023-12-04T05:39:59.719236+0000","last_unstale":"2023-12-04T05:40:01.502157+0000","last_undegraded":"2023-12-04T05:40:01.502157+0000","last_fullsized":"2023-12-04T05:40:01.502157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:23:39.909425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503754+0000","last_change":"2023-12-04T05:39:59.732301+0000","last_active":"2023-12-04T05:40:01.503754+0000","last_peered":"2023-12-04T05:40:01.503754+0000","last_clean":"2023-12-04T05:40:01.503754+0000","last_became_active":"2023-12-04T05:39:59.731561+0000","last_became_peered":"2023-12-04T05:39:59.731561+0000","last_unstale":"2023-12-04T05:40:01.503754+0000","last_undegraded":"2023-12-04T05:40:01.503754+0000","last_fullsized":"2023-12-04T05:40:01.503754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:49:40.179867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501688+0000","last_change":"2023-12-04T05:39:59.721633+0000","last_active":"2023-12-04T05:40:01.501688+0000","last_peered":"2023-12-04T05:40:01.501688+0000","last_clean":"2023-12-04T05:40:01.501688+0000","last_became_active":"2023-12-04T05:39:59.720846+0000","last_became_peered":"2023-12-04T05:39:59.720846+0000","last_unstale":"2023-12-04T05:40:01.501688+0000","last_undegraded":"2023-12-04T05:40:01.501688+0000","last_fullsized":"2023-12-04T05:40:01.501688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:44:34.893307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501835+0000","last_change":"2023-12-04T05:39:59.721173+0000","last_active":"2023-12-04T05:40:01.501835+0000","last_peered":"2023-12-04T05:40:01.501835+0000","last_clean":"2023-12-04T05:40:01.501835+0000","last_became_active":"2023-12-04T05:39:59.720371+0000","last_became_peered":"2023-12-04T05:39:59.720371+0000","last_unstale":"2023-12-04T05:40:01.501835+0000","last_undegraded":"2023-12-04T05:40:01.501835+0000","last_fullsized":"2023-12-04T05:40:01.501835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:12:23.683644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682272+0000","last_change":"2023-12-04T05:39:59.710269+0000","last_active":"2023-12-04T05:40:00.682272+0000","last_peered":"2023-12-04T05:40:00.682272+0000","last_clean":"2023-12-04T05:40:00.682272+0000","last_became_active":"2023-12-04T05:39:59.709565+0000","last_became_peered":"2023-12-04T05:39:59.709565+0000","last_unstale":"2023-12-04T05:40:00.682272+0000","last_undegraded":"2023-12-04T05:40:00.682272+0000","last_fullsized":"2023-12-04T05:40:00.682272+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:11:13.222803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501884+0000","last_change":"2023-12-04T05:39:59.710484+0000","last_active":"2023-12-04T05:40:01.501884+0000","last_peered":"2023-12-04T05:40:01.501884+0000","last_clean":"2023-12-04T05:40:01.501884+0000","last_became_active":"2023-12-04T05:39:59.709623+0000","last_became_peered":"2023-12-04T05:39:59.709623+0000","last_unstale":"2023-12-04T05:40:01.501884+0000","last_undegraded":"2023-12-04T05:40:01.501884+0000","last_fullsized":"2023-12-04T05:40:01.501884+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:40:05.852673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.500880+0000","last_change":"2023-12-04T05:39:59.715685+0000","last_active":"2023-12-04T05:40:01.500880+0000","last_peered":"2023-12-04T05:40:01.500880+0000","last_clean":"2023-12-04T05:40:01.500880+0000","last_became_active":"2023-12-04T05:39:59.714890+0000","last_became_peered":"2023-12-04T05:39:59.714890+0000","last_unstale":"2023-12-04T05:40:01.500880+0000","last_undegraded":"2023-12-04T05:40:01.500880+0000","last_fullsized":"2023-12-04T05:40:01.500880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:59:34.243902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908703+0000","last_change":"2023-12-04T05:39:59.688405+0000","last_active":"2023-12-04T05:40:00.908703+0000","last_peered":"2023-12-04T05:40:00.908703+0000","last_clean":"2023-12-04T05:40:00.908703+0000","last_became_active":"2023-12-04T05:39:59.687603+0000","last_became_peered":"2023-12-04T05:39:59.687603+0000","last_unstale":"2023-12-04T05:40:00.908703+0000","last_undegraded":"2023-12-04T05:40:00.908703+0000","last_fullsized":"2023-12-04T05:40:00.908703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:44:51.269794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681203+0000","last_change":"2023-12-04T05:39:59.718543+0000","last_active":"2023-12-04T05:40:00.681203+0000","last_peered":"2023-12-04T05:40:00.681203+0000","last_clean":"2023-12-04T05:40:00.681203+0000","last_became_active":"2023-12-04T05:39:59.717826+0000","last_became_peered":"2023-12-04T05:39:59.717826+0000","last_unstale":"2023-12-04T05:40:00.681203+0000","last_undegraded":"2023-12-04T05:40:00.681203+0000","last_fullsized":"2023-12-04T05:40:00.681203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:31:03.248114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.684087+0000","last_change":"2023-12-04T05:39:59.724278+0000","last_active":"2023-12-04T05:40:00.684087+0000","last_peered":"2023-12-04T05:40:00.684087+0000","last_clean":"2023-12-04T05:40:00.684087+0000","last_became_active":"2023-12-04T05:39:59.723384+0000","last_became_peered":"2023-12-04T05:39:59.723384+0000","last_unstale":"2023-12-04T05:40:00.684087+0000","last_undegraded":"2023-12-04T05:40:00.684087+0000","last_fullsized":"2023-12-04T05:40:00.684087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:07:20.576260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.679022+0000","last_change":"2023-12-04T05:39:59.711802+0000","last_active":"2023-12-04T05:40:00.679022+0000","last_peered":"2023-12-04T05:40:00.679022+0000","last_clean":"2023-12-04T05:40:00.679022+0000","last_became_active":"2023-12-04T05:39:59.711044+0000","last_became_peered":"2023-12-04T05:39:59.711044+0000","last_unstale":"2023-12-04T05:40:00.679022+0000","last_undegraded":"2023-12-04T05:40:00.679022+0000","last_fullsized":"2023-12-04T05:40:00.679022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:42:28.221010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503300+0000","last_change":"2023-12-04T05:39:59.725226+0000","last_active":"2023-12-04T05:40:01.503300+0000","last_peered":"2023-12-04T05:40:01.503300+0000","last_clean":"2023-12-04T05:40:01.503300+0000","last_became_active":"2023-12-04T05:39:59.724334+0000","last_became_peered":"2023-12-04T05:39:59.724334+0000","last_unstale":"2023-12-04T05:40:01.503300+0000","last_undegraded":"2023-12-04T05:40:01.503300+0000","last_fullsized":"2023-12-04T05:40:01.503300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:48:30.439285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501624+0000","last_change":"2023-12-04T05:39:59.729858+0000","last_active":"2023-12-04T05:40:01.501624+0000","last_peered":"2023-12-04T05:40:01.501624+0000","last_clean":"2023-12-04T05:40:01.501624+0000","last_became_active":"2023-12-04T05:39:59.728989+0000","last_became_peered":"2023-12-04T05:39:59.728989+0000","last_unstale":"2023-12-04T05:40:01.501624+0000","last_undegraded":"2023-12-04T05:40:01.501624+0000","last_fullsized":"2023-12-04T05:40:01.501624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:42:01.099743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501616+0000","last_change":"2023-12-04T05:39:59.724022+0000","last_active":"2023-12-04T05:40:01.501616+0000","last_peered":"2023-12-04T05:40:01.501616+0000","last_clean":"2023-12-04T05:40:01.501616+0000","last_became_active":"2023-12-04T05:39:59.723120+0000","last_became_peered":"2023-12-04T05:39:59.723120+0000","last_unstale":"2023-12-04T05:40:01.501616+0000","last_undegraded":"2023-12-04T05:40:01.501616+0000","last_fullsized":"2023-12-04T05:40:01.501616+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:13:50.048757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683322+0000","last_change":"2023-12-04T05:39:59.714806+0000","last_active":"2023-12-04T05:40:00.683322+0000","last_peered":"2023-12-04T05:40:00.683322+0000","last_clean":"2023-12-04T05:40:00.683322+0000","last_became_active":"2023-12-04T05:39:59.713917+0000","last_became_peered":"2023-12-04T05:39:59.713917+0000","last_unstale":"2023-12-04T05:40:00.683322+0000","last_undegraded":"2023-12-04T05:40:00.683322+0000","last_fullsized":"2023-12-04T05:40:00.683322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:18:03.128562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501966+0000","last_change":"2023-12-04T05:39:59.729098+0000","last_active":"2023-12-04T05:40:01.501966+0000","last_peered":"2023-12-04T05:40:01.501966+0000","last_clean":"2023-12-04T05:40:01.501966+0000","last_became_active":"2023-12-04T05:39:59.728449+0000","last_became_peered":"2023-12-04T05:39:59.728449+0000","last_unstale":"2023-12-04T05:40:01.501966+0000","last_undegraded":"2023-12-04T05:40:01.501966+0000","last_fullsized":"2023-12-04T05:40:01.501966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:25:12.098431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682551+0000","last_change":"2023-12-04T05:39:59.708300+0000","last_active":"2023-12-04T05:40:00.682551+0000","last_peered":"2023-12-04T05:40:00.682551+0000","last_clean":"2023-12-04T05:40:00.682551+0000","last_became_active":"2023-12-04T05:39:59.707530+0000","last_became_peered":"2023-12-04T05:39:59.707530+0000","last_unstale":"2023-12-04T05:40:00.682551+0000","last_undegraded":"2023-12-04T05:40:00.682551+0000","last_fullsized":"2023-12-04T05:40:00.682551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:11:40.053312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502562+0000","last_change":"2023-12-04T05:39:59.729406+0000","last_active":"2023-12-04T05:40:01.502562+0000","last_peered":"2023-12-04T05:40:01.502562+0000","last_clean":"2023-12-04T05:40:01.502562+0000","last_became_active":"2023-12-04T05:39:59.728677+0000","last_became_peered":"2023-12-04T05:39:59.728677+0000","last_unstale":"2023-12-04T05:40:01.502562+0000","last_undegraded":"2023-12-04T05:40:01.502562+0000","last_fullsized":"2023-12-04T05:40:01.502562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:03:26.954387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501922+0000","last_change":"2023-12-04T05:39:59.733628+0000","last_active":"2023-12-04T05:40:01.501922+0000","last_peered":"2023-12-04T05:40:01.501922+0000","last_clean":"2023-12-04T05:40:01.501922+0000","last_became_active":"2023-12-04T05:39:59.726137+0000","last_became_peered":"2023-12-04T05:39:59.726137+0000","last_unstale":"2023-12-04T05:40:01.501922+0000","last_undegraded":"2023-12-04T05:40:01.501922+0000","last_fullsized":"2023-12-04T05:40:01.501922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:49:17.630085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908788+0000","last_change":"2023-12-04T05:39:59.727169+0000","last_active":"2023-12-04T05:40:00.908788+0000","last_peered":"2023-12-04T05:40:00.908788+0000","last_clean":"2023-12-04T05:40:00.908788+0000","last_became_active":"2023-12-04T05:39:59.725378+0000","last_became_peered":"2023-12-04T05:39:59.725378+0000","last_unstale":"2023-12-04T05:40:00.908788+0000","last_undegraded":"2023-12-04T05:40:00.908788+0000","last_fullsized":"2023-12-04T05:40:00.908788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:23:21.007397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502427+0000","last_change":"2023-12-04T05:39:59.729516+0000","last_active":"2023-12-04T05:40:01.502427+0000","last_peered":"2023-12-04T05:40:01.502427+0000","last_clean":"2023-12-04T05:40:01.502427+0000","last_became_active":"2023-12-04T05:39:59.728835+0000","last_became_peered":"2023-12-04T05:39:59.728835+0000","last_unstale":"2023-12-04T05:40:01.502427+0000","last_undegraded":"2023-12-04T05:40:01.502427+0000","last_fullsized":"2023-12-04T05:40:01.502427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:37:37.490362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501933+0000","last_change":"2023-12-04T05:39:59.724592+0000","last_active":"2023-12-04T05:40:01.501933+0000","last_peered":"2023-12-04T05:40:01.501933+0000","last_clean":"2023-12-04T05:40:01.501933+0000","last_became_active":"2023-12-04T05:39:59.723876+0000","last_became_peered":"2023-12-04T05:39:59.723876+0000","last_unstale":"2023-12-04T05:40:01.501933+0000","last_undegraded":"2023-12-04T05:40:01.501933+0000","last_fullsized":"2023-12-04T05:40:01.501933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:09:26.625728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683703+0000","last_change":"2023-12-04T05:39:59.726540+0000","last_active":"2023-12-04T05:40:00.683703+0000","last_peered":"2023-12-04T05:40:00.683703+0000","last_clean":"2023-12-04T05:40:00.683703+0000","last_became_active":"2023-12-04T05:39:59.725781+0000","last_became_peered":"2023-12-04T05:39:59.725781+0000","last_unstale":"2023-12-04T05:40:00.683703+0000","last_undegraded":"2023-12-04T05:40:00.683703+0000","last_fullsized":"2023-12-04T05:40:00.683703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:59:22.013410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.678751+0000","last_change":"2023-12-04T05:39:59.708376+0000","last_active":"2023-12-04T05:40:00.678751+0000","last_peered":"2023-12-04T05:40:00.678751+0000","last_clean":"2023-12-04T05:40:00.678751+0000","last_became_active":"2023-12-04T05:39:59.707604+0000","last_became_peered":"2023-12-04T05:39:59.707604+0000","last_unstale":"2023-12-04T05:40:00.678751+0000","last_undegraded":"2023-12-04T05:40:00.678751+0000","last_fullsized":"2023-12-04T05:40:00.678751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:00:23.385619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501476+0000","last_change":"2023-12-04T05:39:59.720510+0000","last_active":"2023-12-04T05:40:01.501476+0000","last_peered":"2023-12-04T05:40:01.501476+0000","last_clean":"2023-12-04T05:40:01.501476+0000","last_became_active":"2023-12-04T05:39:59.719704+0000","last_became_peered":"2023-12-04T05:39:59.719704+0000","last_unstale":"2023-12-04T05:40:01.501476+0000","last_undegraded":"2023-12-04T05:40:01.501476+0000","last_fullsized":"2023-12-04T05:40:01.501476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:39:15.714817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681790+0000","last_change":"2023-12-04T05:39:59.711839+0000","last_active":"2023-12-04T05:40:00.681790+0000","last_peered":"2023-12-04T05:40:00.681790+0000","last_clean":"2023-12-04T05:40:00.681790+0000","last_became_active":"2023-12-04T05:39:59.711113+0000","last_became_peered":"2023-12-04T05:39:59.711113+0000","last_unstale":"2023-12-04T05:40:00.681790+0000","last_undegraded":"2023-12-04T05:40:00.681790+0000","last_fullsized":"2023-12-04T05:40:00.681790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:21:17.569525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:10.823687+0000","last_change":"2023-12-04T05:39:58.750554+0000","last_active":"2023-12-04T05:40:10.823687+0000","last_peered":"2023-12-04T05:40:10.823687+0000","last_clean":"2023-12-04T05:40:10.823687+0000","last_became_active":"2023-12-04T05:39:58.750227+0000","last_became_peered":"2023-12-04T05:39:58.750227+0000","last_unstale":"2023-12-04T05:40:10.823687+0000","last_undegraded":"2023-12-04T05:40:10.823687+0000","last_fullsized":"2023-12-04T05:40:10.823687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:13:23.997255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502276+0000","last_change":"2023-12-04T05:39:59.731582+0000","last_active":"2023-12-04T05:40:01.502276+0000","last_peered":"2023-12-04T05:40:01.502276+0000","last_clean":"2023-12-04T05:40:01.502276+0000","last_became_active":"2023-12-04T05:39:59.730822+0000","last_became_peered":"2023-12-04T05:39:59.730822+0000","last_unstale":"2023-12-04T05:40:01.502276+0000","last_undegraded":"2023-12-04T05:40:01.502276+0000","last_fullsized":"2023-12-04T05:40:01.502276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:10:55.996104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.909876+0000","last_change":"2023-12-04T05:39:58.740011+0000","last_active":"2023-12-04T05:40:22.909876+0000","last_peered":"2023-12-04T05:40:22.909876+0000","last_clean":"2023-12-04T05:40:22.909876+0000","last_became_active":"2023-12-04T05:39:58.739395+0000","last_became_peered":"2023-12-04T05:39:58.739395+0000","last_unstale":"2023-12-04T05:40:22.909876+0000","last_undegraded":"2023-12-04T05:40:22.909876+0000","last_fullsized":"2023-12-04T05:40:22.909876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:17:48.107311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:16.865405+0000","last_change":"2023-12-04T05:39:58.730186+0000","last_active":"2023-12-04T05:40:16.865405+0000","last_peered":"2023-12-04T05:40:16.865405+0000","last_clean":"2023-12-04T05:40:16.865405+0000","last_became_active":"2023-12-04T05:39:58.729411+0000","last_became_peered":"2023-12-04T05:39:58.729411+0000","last_unstale":"2023-12-04T05:40:16.865405+0000","last_undegraded":"2023-12-04T05:40:16.865405+0000","last_fullsized":"2023-12-04T05:40:16.865405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:39:05.463010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502530+0000","last_change":"2023-12-04T05:39:59.715596+0000","last_active":"2023-12-04T05:40:01.502530+0000","last_peered":"2023-12-04T05:40:01.502530+0000","last_clean":"2023-12-04T05:40:01.502530+0000","last_became_active":"2023-12-04T05:39:59.714842+0000","last_became_peered":"2023-12-04T05:39:59.714842+0000","last_unstale":"2023-12-04T05:40:01.502530+0000","last_undegraded":"2023-12-04T05:40:01.502530+0000","last_fullsized":"2023-12-04T05:40:01.502530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:51:10.874053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.903050+0000","last_change":"2023-12-04T05:39:58.722113+0000","last_active":"2023-12-04T05:40:22.903050+0000","last_peered":"2023-12-04T05:40:22.903050+0000","last_clean":"2023-12-04T05:40:22.903050+0000","last_became_active":"2023-12-04T05:39:58.721346+0000","last_became_peered":"2023-12-04T05:39:58.721346+0000","last_unstale":"2023-12-04T05:40:22.903050+0000","last_undegraded":"2023-12-04T05:40:22.903050+0000","last_fullsized":"2023-12-04T05:40:22.903050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:30:11.578047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908461+0000","last_change":"2023-12-04T05:39:59.708575+0000","last_active":"2023-12-04T05:40:00.908461+0000","last_peered":"2023-12-04T05:40:00.908461+0000","last_clean":"2023-12-04T05:40:00.908461+0000","last_became_active":"2023-12-04T05:39:59.707753+0000","last_became_peered":"2023-12-04T05:39:59.707753+0000","last_unstale":"2023-12-04T05:40:00.908461+0000","last_undegraded":"2023-12-04T05:40:00.908461+0000","last_fullsized":"2023-12-04T05:40:00.908461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:47:58.349368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:10.822709+0000","last_change":"2023-12-04T05:39:58.734739+0000","last_active":"2023-12-04T05:40:10.822709+0000","last_peered":"2023-12-04T05:40:10.822709+0000","last_clean":"2023-12-04T05:40:10.822709+0000","last_became_active":"2023-12-04T05:39:58.733897+0000","last_became_peered":"2023-12-04T05:39:58.733897+0000","last_unstale":"2023-12-04T05:40:10.822709+0000","last_undegraded":"2023-12-04T05:40:10.822709+0000","last_fullsized":"2023-12-04T05:40:10.822709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:52:15.157364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683392+0000","last_change":"2023-12-04T05:39:59.715796+0000","last_active":"2023-12-04T05:40:00.683392+0000","last_peered":"2023-12-04T05:40:00.683392+0000","last_clean":"2023-12-04T05:40:00.683392+0000","last_became_active":"2023-12-04T05:39:59.714951+0000","last_became_peered":"2023-12-04T05:39:59.714951+0000","last_unstale":"2023-12-04T05:40:00.683392+0000","last_undegraded":"2023-12-04T05:40:00.683392+0000","last_fullsized":"2023-12-04T05:40:00.683392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:31:12.475136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.933245+0000","last_change":"2023-12-04T05:39:58.725387+0000","last_active":"2023-12-04T05:40:28.933245+0000","last_peered":"2023-12-04T05:40:28.933245+0000","last_clean":"2023-12-04T05:40:28.933245+0000","last_became_active":"2023-12-04T05:39:58.724672+0000","last_became_peered":"2023-12-04T05:39:58.724672+0000","last_unstale":"2023-12-04T05:40:28.933245+0000","last_undegraded":"2023-12-04T05:40:28.933245+0000","last_fullsized":"2023-12-04T05:40:28.933245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:19:43.112172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503316+0000","last_change":"2023-12-04T05:39:59.718789+0000","last_active":"2023-12-04T05:40:01.503316+0000","last_peered":"2023-12-04T05:40:01.503316+0000","last_clean":"2023-12-04T05:40:01.503316+0000","last_became_active":"2023-12-04T05:39:59.717927+0000","last_became_peered":"2023-12-04T05:39:59.717927+0000","last_unstale":"2023-12-04T05:40:01.503316+0000","last_undegraded":"2023-12-04T05:40:01.503316+0000","last_fullsized":"2023-12-04T05:40:01.503316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:43:08.771461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:10.820444+0000","last_change":"2023-12-04T05:39:58.731710+0000","last_active":"2023-12-04T05:40:10.820444+0000","last_peered":"2023-12-04T05:40:10.820444+0000","last_clean":"2023-12-04T05:40:10.820444+0000","last_became_active":"2023-12-04T05:39:58.731373+0000","last_became_peered":"2023-12-04T05:39:58.731373+0000","last_unstale":"2023-12-04T05:40:10.820444+0000","last_undegraded":"2023-12-04T05:40:10.820444+0000","last_fullsized":"2023-12-04T05:40:10.820444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:03:37.899195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.682594+0000","last_change":"2023-12-04T05:39:59.725444+0000","last_active":"2023-12-04T05:40:00.682594+0000","last_peered":"2023-12-04T05:40:00.682594+0000","last_clean":"2023-12-04T05:40:00.682594+0000","last_became_active":"2023-12-04T05:39:59.724620+0000","last_became_peered":"2023-12-04T05:39:59.724620+0000","last_unstale":"2023-12-04T05:40:00.682594+0000","last_undegraded":"2023-12-04T05:40:00.682594+0000","last_fullsized":"2023-12-04T05:40:00.682594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:11:17.092060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.941936+0000","last_change":"2023-12-04T05:39:58.730135+0000","last_active":"2023-12-04T05:40:28.941936+0000","last_peered":"2023-12-04T05:40:28.941936+0000","last_clean":"2023-12-04T05:40:28.941936+0000","last_became_active":"2023-12-04T05:39:58.729295+0000","last_became_peered":"2023-12-04T05:39:58.729295+0000","last_unstale":"2023-12-04T05:40:28.941936+0000","last_undegraded":"2023-12-04T05:40:28.941936+0000","last_fullsized":"2023-12-04T05:40:28.941936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:24:42.912063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502658+0000","last_change":"2023-12-04T05:39:59.731327+0000","last_active":"2023-12-04T05:40:01.502658+0000","last_peered":"2023-12-04T05:40:01.502658+0000","last_clean":"2023-12-04T05:40:01.502658+0000","last_became_active":"2023-12-04T05:39:59.730427+0000","last_became_peered":"2023-12-04T05:39:59.730427+0000","last_unstale":"2023-12-04T05:40:01.502658+0000","last_undegraded":"2023-12-04T05:40:01.502658+0000","last_fullsized":"2023-12-04T05:40:01.502658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:34:45.211951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'9","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.937241+0000","last_change":"2023-12-04T05:39:58.711690+0000","last_active":"2023-12-04T05:40:28.937241+0000","last_peered":"2023-12-04T05:40:28.937241+0000","last_clean":"2023-12-04T05:40:28.937241+0000","last_became_active":"2023-12-04T05:39:58.711305+0000","last_became_peered":"2023-12-04T05:39:58.711305+0000","last_unstale":"2023-12-04T05:40:28.937241+0000","last_undegraded":"2023-12-04T05:40:28.937241+0000","last_fullsized":"2023-12-04T05:40:28.937241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:33:02.159265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501048+0000","last_change":"2023-12-04T05:39:59.702650+0000","last_active":"2023-12-04T05:40:01.501048+0000","last_peered":"2023-12-04T05:40:01.501048+0000","last_clean":"2023-12-04T05:40:01.501048+0000","last_became_active":"2023-12-04T05:39:59.701827+0000","last_became_peered":"2023-12-04T05:39:59.701827+0000","last_unstale":"2023-12-04T05:40:01.501048+0000","last_undegraded":"2023-12-04T05:40:01.501048+0000","last_fullsized":"2023-12-04T05:40:01.501048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:46:45.006062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.902757+0000","last_change":"2023-12-04T05:39:58.729371+0000","last_active":"2023-12-04T05:40:22.902757+0000","last_peered":"2023-12-04T05:40:22.902757+0000","last_clean":"2023-12-04T05:40:22.902757+0000","last_became_active":"2023-12-04T05:39:58.727781+0000","last_became_peered":"2023-12-04T05:39:58.727781+0000","last_unstale":"2023-12-04T05:40:22.902757+0000","last_undegraded":"2023-12-04T05:40:22.902757+0000","last_fullsized":"2023-12-04T05:40:22.902757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:37:29.707707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502646+0000","last_change":"2023-12-04T05:39:59.742080+0000","last_active":"2023-12-04T05:40:01.502646+0000","last_peered":"2023-12-04T05:40:01.502646+0000","last_clean":"2023-12-04T05:40:01.502646+0000","last_became_active":"2023-12-04T05:39:59.741440+0000","last_became_peered":"2023-12-04T05:39:59.741440+0000","last_unstale":"2023-12-04T05:40:01.502646+0000","last_undegraded":"2023-12-04T05:40:01.502646+0000","last_fullsized":"2023-12-04T05:40:01.502646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:29:21.379583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.908886+0000","last_change":"2023-12-04T05:39:58.737926+0000","last_active":"2023-12-04T05:40:22.908886+0000","last_peered":"2023-12-04T05:40:22.908886+0000","last_clean":"2023-12-04T05:40:22.908886+0000","last_became_active":"2023-12-04T05:39:58.737403+0000","last_became_peered":"2023-12-04T05:39:58.737403+0000","last_unstale":"2023-12-04T05:40:22.908886+0000","last_undegraded":"2023-12-04T05:40:22.908886+0000","last_fullsized":"2023-12-04T05:40:22.908886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:10:59.035282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501005+0000","last_change":"2023-12-04T05:39:59.710310+0000","last_active":"2023-12-04T05:40:01.501005+0000","last_peered":"2023-12-04T05:40:01.501005+0000","last_clean":"2023-12-04T05:40:01.501005+0000","last_became_active":"2023-12-04T05:39:59.709497+0000","last_became_peered":"2023-12-04T05:39:59.709497+0000","last_unstale":"2023-12-04T05:40:01.501005+0000","last_undegraded":"2023-12-04T05:40:01.501005+0000","last_fullsized":"2023-12-04T05:40:01.501005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:43:43.708311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'17","reported_seq":29,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.899860+0000","last_change":"2023-12-04T05:39:58.728046+0000","last_active":"2023-12-04T05:40:22.899860+0000","last_peered":"2023-12-04T05:40:22.899860+0000","last_clean":"2023-12-04T05:40:22.899860+0000","last_became_active":"2023-12-04T05:39:58.726747+0000","last_became_peered":"2023-12-04T05:39:58.726747+0000","last_unstale":"2023-12-04T05:40:22.899860+0000","last_undegraded":"2023-12-04T05:40:22.899860+0000","last_fullsized":"2023-12-04T05:40:22.899860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:45:11.024395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501689+0000","last_change":"2023-12-04T05:39:59.741770+0000","last_active":"2023-12-04T05:40:01.501689+0000","last_peered":"2023-12-04T05:40:01.501689+0000","last_clean":"2023-12-04T05:40:01.501689+0000","last_became_active":"2023-12-04T05:39:59.741091+0000","last_became_peered":"2023-12-04T05:39:59.741091+0000","last_unstale":"2023-12-04T05:40:01.501689+0000","last_undegraded":"2023-12-04T05:40:01.501689+0000","last_fullsized":"2023-12-04T05:40:01.501689+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:59:20.747612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.772742+0000","last_change":"2023-12-04T05:39:58.712970+0000","last_active":"2023-12-04T05:40:04.772742+0000","last_peered":"2023-12-04T05:40:04.772742+0000","last_clean":"2023-12-04T05:40:04.772742+0000","last_became_active":"2023-12-04T05:39:58.711759+0000","last_became_peered":"2023-12-04T05:39:58.711759+0000","last_unstale":"2023-12-04T05:40:04.772742+0000","last_undegraded":"2023-12-04T05:40:04.772742+0000","last_fullsized":"2023-12-04T05:40:04.772742+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:02:00.957563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503407+0000","last_change":"2023-12-04T05:39:59.742244+0000","last_active":"2023-12-04T05:40:01.503407+0000","last_peered":"2023-12-04T05:40:01.503407+0000","last_clean":"2023-12-04T05:40:01.503407+0000","last_became_active":"2023-12-04T05:39:59.741379+0000","last_became_peered":"2023-12-04T05:39:59.741379+0000","last_unstale":"2023-12-04T05:40:01.503407+0000","last_undegraded":"2023-12-04T05:40:01.503407+0000","last_fullsized":"2023-12-04T05:40:01.503407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:07:57.685895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.934917+0000","last_change":"2023-12-04T05:39:58.729723+0000","last_active":"2023-12-04T05:40:28.934917+0000","last_peered":"2023-12-04T05:40:28.934917+0000","last_clean":"2023-12-04T05:40:28.934917+0000","last_became_active":"2023-12-04T05:39:58.729410+0000","last_became_peered":"2023-12-04T05:39:58.729410+0000","last_unstale":"2023-12-04T05:40:28.934917+0000","last_undegraded":"2023-12-04T05:40:28.934917+0000","last_fullsized":"2023-12-04T05:40:28.934917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:59:10.799774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503914+0000","last_change":"2023-12-04T05:39:59.741718+0000","last_active":"2023-12-04T05:40:01.503914+0000","last_peered":"2023-12-04T05:40:01.503914+0000","last_clean":"2023-12-04T05:40:01.503914+0000","last_became_active":"2023-12-04T05:39:59.741101+0000","last_became_peered":"2023-12-04T05:39:59.741101+0000","last_unstale":"2023-12-04T05:40:01.503914+0000","last_undegraded":"2023-12-04T05:40:01.503914+0000","last_fullsized":"2023-12-04T05:40:01.503914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:41:03.969161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.900442+0000","last_change":"2023-12-04T05:39:58.737224+0000","last_active":"2023-12-04T05:40:22.900442+0000","last_peered":"2023-12-04T05:40:22.900442+0000","last_clean":"2023-12-04T05:40:22.900442+0000","last_became_active":"2023-12-04T05:39:58.736753+0000","last_became_peered":"2023-12-04T05:39:58.736753+0000","last_unstale":"2023-12-04T05:40:22.900442+0000","last_undegraded":"2023-12-04T05:40:22.900442+0000","last_fullsized":"2023-12-04T05:40:22.900442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:33:34.027796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683405+0000","last_change":"2023-12-04T05:39:59.726776+0000","last_active":"2023-12-04T05:40:00.683405+0000","last_peered":"2023-12-04T05:40:00.683405+0000","last_clean":"2023-12-04T05:40:00.683405+0000","last_became_active":"2023-12-04T05:39:59.725991+0000","last_became_peered":"2023-12-04T05:39:59.725991+0000","last_unstale":"2023-12-04T05:40:00.683405+0000","last_undegraded":"2023-12-04T05:40:00.683405+0000","last_fullsized":"2023-12-04T05:40:00.683405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:36:02.168898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.905810+0000","last_change":"2023-12-04T05:39:58.732942+0000","last_active":"2023-12-04T05:40:22.905810+0000","last_peered":"2023-12-04T05:40:22.905810+0000","last_clean":"2023-12-04T05:40:22.905810+0000","last_became_active":"2023-12-04T05:39:58.732401+0000","last_became_peered":"2023-12-04T05:39:58.732401+0000","last_unstale":"2023-12-04T05:40:22.905810+0000","last_undegraded":"2023-12-04T05:40:22.905810+0000","last_fullsized":"2023-12-04T05:40:22.905810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:23:13.676061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503215+0000","last_change":"2023-12-04T05:39:59.730768+0000","last_active":"2023-12-04T05:40:01.503215+0000","last_peered":"2023-12-04T05:40:01.503215+0000","last_clean":"2023-12-04T05:40:01.503215+0000","last_became_active":"2023-12-04T05:39:59.730207+0000","last_became_peered":"2023-12-04T05:39:59.730207+0000","last_unstale":"2023-12-04T05:40:01.503215+0000","last_undegraded":"2023-12-04T05:40:01.503215+0000","last_fullsized":"2023-12-04T05:40:01.503215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:25:19.473188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T05:39:59.705581+0000","last_change":"2023-12-04T05:39:49.314208+0000","last_active":"2023-12-04T05:39:59.705581+0000","last_peered":"2023-12-04T05:39:59.705581+0000","last_clean":"2023-12-04T05:39:59.705581+0000","last_became_active":"2023-12-04T05:39:49.312262+0000","last_became_peered":"2023-12-04T05:39:49.312262+0000","last_unstale":"2023-12-04T05:39:59.705581+0000","last_undegraded":"2023-12-04T05:39:59.705581+0000","last_fullsized":"2023-12-04T05:39:59.705581+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:48.242612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:48.242612+0000","last_clean_scrub_stamp":"2023-12-04T05:39:48.242612+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:19:26.511345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.907597+0000","last_change":"2023-12-04T05:39:58.725538+0000","last_active":"2023-12-04T05:40:22.907597+0000","last_peered":"2023-12-04T05:40:22.907597+0000","last_clean":"2023-12-04T05:40:22.907597+0000","last_became_active":"2023-12-04T05:39:58.724839+0000","last_became_peered":"2023-12-04T05:39:58.724839+0000","last_unstale":"2023-12-04T05:40:22.907597+0000","last_undegraded":"2023-12-04T05:40:22.907597+0000","last_fullsized":"2023-12-04T05:40:22.907597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:34:48.521117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501800+0000","last_change":"2023-12-04T05:39:59.719475+0000","last_active":"2023-12-04T05:40:01.501800+0000","last_peered":"2023-12-04T05:40:01.501800+0000","last_clean":"2023-12-04T05:40:01.501800+0000","last_became_active":"2023-12-04T05:39:59.718727+0000","last_became_peered":"2023-12-04T05:39:59.718727+0000","last_unstale":"2023-12-04T05:40:01.501800+0000","last_undegraded":"2023-12-04T05:40:01.501800+0000","last_fullsized":"2023-12-04T05:40:01.501800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:53:35.787086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:16.863882+0000","last_change":"2023-12-04T05:39:58.751135+0000","last_active":"2023-12-04T05:40:16.863882+0000","last_peered":"2023-12-04T05:40:16.863882+0000","last_clean":"2023-12-04T05:40:16.863882+0000","last_became_active":"2023-12-04T05:39:58.750144+0000","last_became_peered":"2023-12-04T05:39:58.750144+0000","last_unstale":"2023-12-04T05:40:16.863882+0000","last_undegraded":"2023-12-04T05:40:16.863882+0000","last_fullsized":"2023-12-04T05:40:16.863882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:31:32.599659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.500789+0000","last_change":"2023-12-04T05:39:59.729585+0000","last_active":"2023-12-04T05:40:01.500789+0000","last_peered":"2023-12-04T05:40:01.500789+0000","last_clean":"2023-12-04T05:40:01.500789+0000","last_became_active":"2023-12-04T05:39:59.728381+0000","last_became_peered":"2023-12-04T05:39:59.728381+0000","last_unstale":"2023-12-04T05:40:01.500789+0000","last_undegraded":"2023-12-04T05:40:01.500789+0000","last_fullsized":"2023-12-04T05:40:01.500789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:35:11.920755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:16.867806+0000","last_change":"2023-12-04T05:39:58.731429+0000","last_active":"2023-12-04T05:40:16.867806+0000","last_peered":"2023-12-04T05:40:16.867806+0000","last_clean":"2023-12-04T05:40:16.867806+0000","last_became_active":"2023-12-04T05:39:58.731049+0000","last_became_peered":"2023-12-04T05:39:58.731049+0000","last_unstale":"2023-12-04T05:40:16.867806+0000","last_undegraded":"2023-12-04T05:40:16.867806+0000","last_fullsized":"2023-12-04T05:40:16.867806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:27:16.863460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908531+0000","last_change":"2023-12-04T05:39:59.701324+0000","last_active":"2023-12-04T05:40:00.908531+0000","last_peered":"2023-12-04T05:40:00.908531+0000","last_clean":"2023-12-04T05:40:00.908531+0000","last_became_active":"2023-12-04T05:39:59.700586+0000","last_became_peered":"2023-12-04T05:39:59.700586+0000","last_unstale":"2023-12-04T05:40:00.908531+0000","last_undegraded":"2023-12-04T05:40:00.908531+0000","last_fullsized":"2023-12-04T05:40:00.908531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:37:17.301688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.937384+0000","last_change":"2023-12-04T05:39:58.729635+0000","last_active":"2023-12-04T05:40:28.937384+0000","last_peered":"2023-12-04T05:40:28.937384+0000","last_clean":"2023-12-04T05:40:28.937384+0000","last_became_active":"2023-12-04T05:39:58.728927+0000","last_became_peered":"2023-12-04T05:39:58.728927+0000","last_unstale":"2023-12-04T05:40:28.937384+0000","last_undegraded":"2023-12-04T05:40:28.937384+0000","last_fullsized":"2023-12-04T05:40:28.937384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:58:46.851065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681790+0000","last_change":"2023-12-04T05:39:59.702840+0000","last_active":"2023-12-04T05:40:00.681790+0000","last_peered":"2023-12-04T05:40:00.681790+0000","last_clean":"2023-12-04T05:40:00.681790+0000","last_became_active":"2023-12-04T05:39:59.701652+0000","last_became_peered":"2023-12-04T05:39:59.701652+0000","last_unstale":"2023-12-04T05:40:00.681790+0000","last_undegraded":"2023-12-04T05:40:00.681790+0000","last_fullsized":"2023-12-04T05:40:00.681790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:44:14.733182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.521524+0000","last_change":"2023-12-04T05:39:58.721834+0000","last_active":"2023-12-04T05:40:01.521524+0000","last_peered":"2023-12-04T05:40:01.521524+0000","last_clean":"2023-12-04T05:40:01.521524+0000","last_became_active":"2023-12-04T05:39:58.721451+0000","last_became_peered":"2023-12-04T05:39:58.721451+0000","last_unstale":"2023-12-04T05:40:01.521524+0000","last_undegraded":"2023-12-04T05:40:01.521524+0000","last_fullsized":"2023-12-04T05:40:01.521524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:15:52.065853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.684129+0000","last_change":"2023-12-04T05:39:59.731053+0000","last_active":"2023-12-04T05:40:00.684129+0000","last_peered":"2023-12-04T05:40:00.684129+0000","last_clean":"2023-12-04T05:40:00.684129+0000","last_became_active":"2023-12-04T05:39:59.729732+0000","last_became_peered":"2023-12-04T05:39:59.729732+0000","last_unstale":"2023-12-04T05:40:00.684129+0000","last_undegraded":"2023-12-04T05:40:00.684129+0000","last_fullsized":"2023-12-04T05:40:00.684129+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:57:38.581311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.903625+0000","last_change":"2023-12-04T05:39:58.751126+0000","last_active":"2023-12-04T05:40:22.903625+0000","last_peered":"2023-12-04T05:40:22.903625+0000","last_clean":"2023-12-04T05:40:22.903625+0000","last_became_active":"2023-12-04T05:39:58.750496+0000","last_became_peered":"2023-12-04T05:39:58.750496+0000","last_unstale":"2023-12-04T05:40:22.903625+0000","last_undegraded":"2023-12-04T05:40:22.903625+0000","last_fullsized":"2023-12-04T05:40:22.903625+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:09:22.333993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.908579+0000","last_change":"2023-12-04T05:39:59.700032+0000","last_active":"2023-12-04T05:40:00.908579+0000","last_peered":"2023-12-04T05:40:00.908579+0000","last_clean":"2023-12-04T05:40:00.908579+0000","last_became_active":"2023-12-04T05:39:59.699488+0000","last_became_peered":"2023-12-04T05:39:59.699488+0000","last_unstale":"2023-12-04T05:40:00.908579+0000","last_undegraded":"2023-12-04T05:40:00.908579+0000","last_fullsized":"2023-12-04T05:40:00.908579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:59:57.153486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.906694+0000","last_change":"2023-12-04T05:39:58.723126+0000","last_active":"2023-12-04T05:40:22.906694+0000","last_peered":"2023-12-04T05:40:22.906694+0000","last_clean":"2023-12-04T05:40:22.906694+0000","last_became_active":"2023-12-04T05:39:58.722766+0000","last_became_peered":"2023-12-04T05:39:58.722766+0000","last_unstale":"2023-12-04T05:40:22.906694+0000","last_undegraded":"2023-12-04T05:40:22.906694+0000","last_fullsized":"2023-12-04T05:40:22.906694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:28:47.171345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502918+0000","last_change":"2023-12-04T05:39:59.724653+0000","last_active":"2023-12-04T05:40:01.502918+0000","last_peered":"2023-12-04T05:40:01.502918+0000","last_clean":"2023-12-04T05:40:01.502918+0000","last_became_active":"2023-12-04T05:39:59.723945+0000","last_became_peered":"2023-12-04T05:39:59.723945+0000","last_unstale":"2023-12-04T05:40:01.502918+0000","last_undegraded":"2023-12-04T05:40:01.502918+0000","last_fullsized":"2023-12-04T05:40:01.502918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T10:13:56.812140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.941971+0000","last_change":"2023-12-04T05:39:58.746966+0000","last_active":"2023-12-04T05:40:28.941971+0000","last_peered":"2023-12-04T05:40:28.941971+0000","last_clean":"2023-12-04T05:40:28.941971+0000","last_became_active":"2023-12-04T05:39:58.746261+0000","last_became_peered":"2023-12-04T05:39:58.746261+0000","last_unstale":"2023-12-04T05:40:28.941971+0000","last_undegraded":"2023-12-04T05:40:28.941971+0000","last_fullsized":"2023-12-04T05:40:28.941971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:57:46.899291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503381+0000","last_change":"2023-12-04T05:39:59.713259+0000","last_active":"2023-12-04T05:40:01.503381+0000","last_peered":"2023-12-04T05:40:01.503381+0000","last_clean":"2023-12-04T05:40:01.503381+0000","last_became_active":"2023-12-04T05:39:59.712523+0000","last_became_peered":"2023-12-04T05:39:59.712523+0000","last_unstale":"2023-12-04T05:40:01.503381+0000","last_undegraded":"2023-12-04T05:40:01.503381+0000","last_fullsized":"2023-12-04T05:40:01.503381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:41:10.001755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:10.821316+0000","last_change":"2023-12-04T05:39:58.750282+0000","last_active":"2023-12-04T05:40:10.821316+0000","last_peered":"2023-12-04T05:40:10.821316+0000","last_clean":"2023-12-04T05:40:10.821316+0000","last_became_active":"2023-12-04T05:39:58.749651+0000","last_became_peered":"2023-12-04T05:39:58.749651+0000","last_unstale":"2023-12-04T05:40:10.821316+0000","last_undegraded":"2023-12-04T05:40:10.821316+0000","last_fullsized":"2023-12-04T05:40:10.821316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:40:55.086649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.681687+0000","last_change":"2023-12-04T05:39:59.704667+0000","last_active":"2023-12-04T05:40:00.681687+0000","last_peered":"2023-12-04T05:40:00.681687+0000","last_clean":"2023-12-04T05:40:00.681687+0000","last_became_active":"2023-12-04T05:39:59.703943+0000","last_became_peered":"2023-12-04T05:39:59.703943+0000","last_unstale":"2023-12-04T05:40:00.681687+0000","last_undegraded":"2023-12-04T05:40:00.681687+0000","last_fullsized":"2023-12-04T05:40:00.681687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:31:53.890532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'7","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.936677+0000","last_change":"2023-12-04T05:39:58.745277+0000","last_active":"2023-12-04T05:40:28.936677+0000","last_peered":"2023-12-04T05:40:28.936677+0000","last_clean":"2023-12-04T05:40:28.936677+0000","last_became_active":"2023-12-04T05:39:58.744842+0000","last_became_peered":"2023-12-04T05:39:58.744842+0000","last_unstale":"2023-12-04T05:40:28.936677+0000","last_undegraded":"2023-12-04T05:40:28.936677+0000","last_fullsized":"2023-12-04T05:40:28.936677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:50:21.084855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502964+0000","last_change":"2023-12-04T05:39:59.714217+0000","last_active":"2023-12-04T05:40:01.502964+0000","last_peered":"2023-12-04T05:40:01.502964+0000","last_clean":"2023-12-04T05:40:01.502964+0000","last_became_active":"2023-12-04T05:39:59.713438+0000","last_became_peered":"2023-12-04T05:39:59.713438+0000","last_unstale":"2023-12-04T05:40:01.502964+0000","last_undegraded":"2023-12-04T05:40:01.502964+0000","last_fullsized":"2023-12-04T05:40:01.502964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:37:30.790544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:22.897924+0000","last_change":"2023-12-04T05:39:58.722048+0000","last_active":"2023-12-04T05:40:22.897924+0000","last_peered":"2023-12-04T05:40:22.897924+0000","last_clean":"2023-12-04T05:40:22.897924+0000","last_became_active":"2023-12-04T05:39:58.721782+0000","last_became_peered":"2023-12-04T05:39:58.721782+0000","last_unstale":"2023-12-04T05:40:22.897924+0000","last_undegraded":"2023-12-04T05:40:22.897924+0000","last_fullsized":"2023-12-04T05:40:22.897924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:38:16.023373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.502222+0000","last_change":"2023-12-04T05:39:59.715313+0000","last_active":"2023-12-04T05:40:01.502222+0000","last_peered":"2023-12-04T05:40:01.502222+0000","last_clean":"2023-12-04T05:40:01.502222+0000","last_became_active":"2023-12-04T05:39:59.714430+0000","last_became_peered":"2023-12-04T05:39:59.714430+0000","last_unstale":"2023-12-04T05:40:01.502222+0000","last_undegraded":"2023-12-04T05:40:01.502222+0000","last_fullsized":"2023-12-04T05:40:01.502222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T06:10:01.533077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:10.813719+0000","last_change":"2023-12-04T05:39:58.750921+0000","last_active":"2023-12-04T05:40:10.813719+0000","last_peered":"2023-12-04T05:40:10.813719+0000","last_clean":"2023-12-04T05:40:10.813719+0000","last_became_active":"2023-12-04T05:39:58.750288+0000","last_became_peered":"2023-12-04T05:39:58.750288+0000","last_unstale":"2023-12-04T05:40:10.813719+0000","last_undegraded":"2023-12-04T05:40:10.813719+0000","last_fullsized":"2023-12-04T05:40:10.813719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:04:26.373696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501279+0000","last_change":"2023-12-04T05:39:59.729260+0000","last_active":"2023-12-04T05:40:01.501279+0000","last_peered":"2023-12-04T05:40:01.501279+0000","last_clean":"2023-12-04T05:40:01.501279+0000","last_became_active":"2023-12-04T05:39:59.728639+0000","last_became_peered":"2023-12-04T05:39:59.728639+0000","last_unstale":"2023-12-04T05:40:01.501279+0000","last_undegraded":"2023-12-04T05:40:01.501279+0000","last_fullsized":"2023-12-04T05:40:01.501279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:21:38.039584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683814+0000","last_change":"2023-12-04T05:39:59.731652+0000","last_active":"2023-12-04T05:40:00.683814+0000","last_peered":"2023-12-04T05:40:00.683814+0000","last_clean":"2023-12-04T05:40:00.683814+0000","last_became_active":"2023-12-04T05:39:59.730714+0000","last_became_peered":"2023-12-04T05:39:59.730714+0000","last_unstale":"2023-12-04T05:40:00.683814+0000","last_undegraded":"2023-12-04T05:40:00.683814+0000","last_fullsized":"2023-12-04T05:40:00.683814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T07:02:27.013221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.936208+0000","last_change":"2023-12-04T05:39:58.751530+0000","last_active":"2023-12-04T05:40:28.936208+0000","last_peered":"2023-12-04T05:40:28.936208+0000","last_clean":"2023-12-04T05:40:28.936208+0000","last_became_active":"2023-12-04T05:39:58.750857+0000","last_became_peered":"2023-12-04T05:39:58.750857+0000","last_unstale":"2023-12-04T05:40:28.936208+0000","last_undegraded":"2023-12-04T05:40:28.936208+0000","last_fullsized":"2023-12-04T05:40:28.936208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:43:36.369813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:00.683195+0000","last_change":"2023-12-04T05:39:59.730535+0000","last_active":"2023-12-04T05:40:00.683195+0000","last_peered":"2023-12-04T05:40:00.683195+0000","last_clean":"2023-12-04T05:40:00.683195+0000","last_became_active":"2023-12-04T05:39:59.729717+0000","last_became_peered":"2023-12-04T05:39:59.729717+0000","last_unstale":"2023-12-04T05:40:00.683195+0000","last_undegraded":"2023-12-04T05:40:00.683195+0000","last_fullsized":"2023-12-04T05:40:00.683195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T09:31:46.120414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'12","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.942457+0000","last_change":"2023-12-04T05:39:58.752193+0000","last_active":"2023-12-04T05:40:28.942457+0000","last_peered":"2023-12-04T05:40:28.942457+0000","last_clean":"2023-12-04T05:40:28.942457+0000","last_became_active":"2023-12-04T05:39:58.751625+0000","last_became_peered":"2023-12-04T05:39:58.751625+0000","last_unstale":"2023-12-04T05:40:28.942457+0000","last_undegraded":"2023-12-04T05:40:28.942457+0000","last_fullsized":"2023-12-04T05:40:28.942457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:05:05.734494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.501167+0000","last_change":"2023-12-04T05:39:59.720197+0000","last_active":"2023-12-04T05:40:01.501167+0000","last_peered":"2023-12-04T05:40:01.501167+0000","last_clean":"2023-12-04T05:40:01.501167+0000","last_became_active":"2023-12-04T05:39:59.718736+0000","last_became_peered":"2023-12-04T05:39:59.718736+0000","last_unstale":"2023-12-04T05:40:01.501167+0000","last_undegraded":"2023-12-04T05:40:01.501167+0000","last_fullsized":"2023-12-04T05:40:01.501167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:41:13.184198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:04.779055+0000","last_change":"2023-12-04T05:39:58.726906+0000","last_active":"2023-12-04T05:40:04.779055+0000","last_peered":"2023-12-04T05:40:04.779055+0000","last_clean":"2023-12-04T05:40:04.779055+0000","last_became_active":"2023-12-04T05:39:58.726415+0000","last_became_peered":"2023-12-04T05:39:58.726415+0000","last_unstale":"2023-12-04T05:40:04.779055+0000","last_undegraded":"2023-12-04T05:40:04.779055+0000","last_fullsized":"2023-12-04T05:40:04.779055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:58:53.715291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:01.503085+0000","last_change":"2023-12-04T05:39:59.711861+0000","last_active":"2023-12-04T05:40:01.503085+0000","last_peered":"2023-12-04T05:40:01.503085+0000","last_clean":"2023-12-04T05:40:01.503085+0000","last_became_active":"2023-12-04T05:39:59.710933+0000","last_became_peered":"2023-12-04T05:39:59.710933+0000","last_unstale":"2023-12-04T05:40:01.503085+0000","last_undegraded":"2023-12-04T05:40:01.503085+0000","last_fullsized":"2023-12-04T05:40:01.503085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:58.684698+0000","last_clean_scrub_stamp":"2023-12-04T05:39:58.684698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T05:40:46.361768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'13","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-04T05:40:28.937128+0000","last_change":"2023-12-04T05:39:58.740704+0000","last_active":"2023-12-04T05:40:28.937128+0000","last_peered":"2023-12-04T05:40:28.937128+0000","last_clean":"2023-12-04T05:40:28.937128+0000","last_became_active":"2023-12-04T05:39:58.740440+0000","last_became_peered":"2023-12-04T05:39:58.740440+0000","last_unstale":"2023-12-04T05:40:28.937128+0000","last_undegraded":"2023-12-04T05:40:28.937128+0000","last_fullsized":"2023-12-04T05:40:28.937128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T05:39:57.675341+0000","last_clean_scrub_stamp":"2023-12-04T05:39:57.675341+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T08:20:18.197917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8034,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":205,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":179,"ondisk_log_size":179,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574862,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":847872,"data_stored":668049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.099}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.202}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]}]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":245760,"data_stored":77983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1539999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1859999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":79513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.871}]}]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":241664,"data_stored":76911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1719999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.246}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.018}]}]},{"osd":3,"up_from":13,"seq":55834574862,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":78017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":77043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574862,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":245760,"data_stored":77113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":839680,"data_stored":666823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-04T05:40:42.797 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T05:40:42.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.954+0000 7f6554895640 1 Processor -- start 2023-12-04T05:40:42.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.954+0000 7f6554895640 1 -- start start 2023-12-04T05:40:42.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f6554895640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f65500c56f0 0x7f6550158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:42.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f6554895640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6550158da0 con 0x7f65500724f0 2023-12-04T05:40:42.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f6554895640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65501583d0 con 0x7f6550072110 2023-12-04T05:40:42.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f6554895640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65500c6170 con 0x7f65500c56f0 2023-12-04T05:40:42.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f6545d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f65500724f0 0x7f6550140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39632/0 (socket says 172.21.15.107:39632) 2023-12-04T05:40:42.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f6545d9b640 1 -- 172.21.15.107:0/4186570222 learned_addr learned my addr 172.21.15.107:0/4186570222 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:42.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654ed9d640 1 --2- 172.21.15.107:0/4186570222 >> v2:172.21.15.195:3300/0 conn(0x7f65500c56f0 0x7f6550158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:42.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654dd9b640 1 -- 172.21.15.107:0/4186570222 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106295558 0 0) 0x7f6538001120 con 0x7f65500724f0 2023-12-04T05:40:42.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654dd9b640 1 -- 172.21.15.107:0/4186570222 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f653c003530 con 0x7f65500724f0 2023-12-04T05:40:42.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654dd9b640 1 -- 172.21.15.107:0/4186570222 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2880256785 0 0) 0x7f653c003530 con 0x7f65500724f0 2023-12-04T05:40:42.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654dd9b640 1 -- 172.21.15.107:0/4186570222 >> v2:172.21.15.195:3300/0 conn(0x7f65500c56f0 msgr2=0x7f6550158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:42.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654dd9b640 1 --2- 172.21.15.107:0/4186570222 >> v2:172.21.15.195:3300/0 conn(0x7f65500c56f0 0x7f6550158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654dd9b640 1 -- 172.21.15.107:0/4186570222 >> v1:172.21.15.195:6789/0 conn(0x7f6550072110 legacy=0x7f65500c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:42.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654dd9b640 1 -- 172.21.15.107:0/4186570222 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6550157da0 con 0x7f65500724f0 2023-12-04T05:40:42.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654dd9b640 1 -- 172.21.15.107:0/4186570222 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f6538001810 con 0x7f65500724f0 2023-12-04T05:40:42.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654dd9b640 1 -- 172.21.15.107:0/4186570222 >> v1:172.21.15.107:6789/0 conn(0x7f65500724f0 legacy=0x7f6550140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:42.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654dd9b640 1 --2- 172.21.15.107:0/4186570222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f653c003980 0x7f653c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:42.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654dd9b640 1 --2- 172.21.15.107:0/4186570222 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f653c0047d0 0x7f653c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:42.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654dd9b640 1 --2- 172.21.15.107:0/4186570222 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f653c005190 0x7f653c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:42.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654dd9b640 1 -- 172.21.15.107:0/4186570222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65500c6170 con 0x7f653c003980 2023-12-04T05:40:42.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654dd9b640 1 -- 172.21.15.107:0/4186570222 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653c00c040 con 0x7f653c0047d0 2023-12-04T05:40:42.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654dd9b640 1 -- 172.21.15.107:0/4186570222 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f653c00c200 con 0x7f653c005190 2023-12-04T05:40:42.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654e59c640 1 --2- 172.21.15.107:0/4186570222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f653c003980 0x7f653c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:42.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f6545d9b640 1 --2- 172.21.15.107:0/4186570222 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f653c0047d0 0x7f653c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:42.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654ed9d640 1 --2- 172.21.15.107:0/4186570222 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f653c005190 0x7f653c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:42.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654e59c640 1 -- 172.21.15.107:0/4186570222 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f653c005190 msgr2=0x7f653c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:42.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654e59c640 1 --2- 172.21.15.107:0/4186570222 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f653c005190 0x7f653c00b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654e59c640 1 -- 172.21.15.107:0/4186570222 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f653c0047d0 msgr2=0x7f653c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:42.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654e59c640 1 --2- 172.21.15.107:0/4186570222 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f653c0047d0 0x7f653c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654e59c640 1 -- 172.21.15.107:0/4186570222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653c003530 con 0x7f653c003980 2023-12-04T05:40:42.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654ed9d640 1 -- 172.21.15.107:0/4186570222 reap_dead start 2023-12-04T05:40:42.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654e59c640 1 --2- 172.21.15.107:0/4186570222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f653c003980 0x7f653c003da0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6540001780 tx=0x7f654001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f583e003751136 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:42.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654dd9b640 1 -- 172.21.15.107:0/4186570222 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65400a4ea0 con 0x7f653c003980 2023-12-04T05:40:42.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654dd9b640 1 -- 172.21.15.107:0/4186570222 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65400a7050 con 0x7f653c003980 2023-12-04T05:40:42.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.958+0000 7f654dd9b640 1 -- 172.21.15.107:0/4186570222 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6540003600 con 0x7f653c003980 2023-12-04T05:40:42.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6554895640 1 -- 172.21.15.107:0/4186570222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f653c003980 msgr2=0x7f653c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:42.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6554895640 1 --2- 172.21.15.107:0/4186570222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f653c003980 0x7f653c003da0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6540001780 tx=0x7f654001c3e0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6554895640 1 -- 172.21.15.107:0/4186570222 shutdown_connections 2023-12-04T05:40:42.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6554895640 1 --2- 172.21.15.107:0/4186570222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f653c003980 0x7f653c003da0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6554895640 1 -- 172.21.15.107:0/4186570222 >> 172.21.15.107:0/4186570222 conn(0x7f65500c7ff0 msgr2=0x7f65500c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:42.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6554895640 1 -- 172.21.15.107:0/4186570222 shutdown_connections 2023-12-04T05:40:42.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6554895640 1 -- 172.21.15.107:0/4186570222 wait complete. 2023-12-04T05:40:42.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6554895640 1 Processor -- start 2023-12-04T05:40:42.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6554895640 1 -- start start 2023-12-04T05:40:42.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6554895640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f653c003980 0x7f655014a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:42.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6554895640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6550158f60 0x7f65501f1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:42.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6554895640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f655006d300 0x7f655014b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:42.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6554895640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6550154180 con 0x7f6550158f60 2023-12-04T05:40:42.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6554895640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65501543d0 con 0x7f655006d300 2023-12-04T05:40:42.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6554895640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65501546b0 con 0x7f653c003980 2023-12-04T05:40:42.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6545d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6550158f60 0x7f65501f1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:42.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f654ed9d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f655006d300 0x7f655014b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:42.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f654e59c640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f653c003980 0x7f655014a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:42.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f654ed9d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f655006d300 0x7f655014b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.107:53168/0 (socket says 172.21.15.107:53168) 2023-12-04T05:40:42.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f654ed9d640 1 -- 172.21.15.107:0/3602664391 learned_addr learned my addr 172.21.15.107:0/3602664391 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:42.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6545d9b640 1 -- 172.21.15.107:0/3602664391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f653c003980 msgr2=0x7f655014a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:42.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6545d9b640 1 --2- 172.21.15.107:0/3602664391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f653c003980 0x7f655014a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6545d9b640 1 -- 172.21.15.107:0/3602664391 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f655006d300 msgr2=0x7f655014b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:42.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6545d9b640 1 --2- 172.21.15.107:0/3602664391 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f655006d300 0x7f655014b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:42.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6545d9b640 1 -- 172.21.15.107:0/3602664391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6550154990 con 0x7f6550158f60 2023-12-04T05:40:42.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6545d9b640 1 --2- 172.21.15.107:0/3602664391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6550158f60 0x7f65501f1180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f655006f1e0 tx=0x7f6538011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:42.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f65477fe640 1 -- 172.21.15.107:0/3602664391 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653800f950 con 0x7f6550158f60 2023-12-04T05:40:42.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f65477fe640 1 -- 172.21.15.107:0/3602664391 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653800faf0 con 0x7f6550158f60 2023-12-04T05:40:42.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f65477fe640 1 -- 172.21.15.107:0/3602664391 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653800fea0 con 0x7f6550158f60 2023-12-04T05:40:42.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6554895640 1 -- 172.21.15.107:0/3602664391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6550154c70 con 0x7f6550158f60 2023-12-04T05:40:42.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.962+0000 7f6554895640 1 -- 172.21.15.107:0/3602664391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6550159430 con 0x7f6550158f60 2023-12-04T05:40:42.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.966+0000 7f6554895640 1 -- 172.21.15.107:0/3602664391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6550155120 con 0x7f6550158f60 2023-12-04T05:40:42.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.966+0000 7f65477fe640 1 -- 172.21.15.107:0/3602664391 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f653801f460 con 0x7f6550158f60 2023-12-04T05:40:42.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.966+0000 7f65477fe640 1 --2- 172.21.15.107:0/3602664391 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f65300668d0 0x7f6530068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:42.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.966+0000 7f654e59c640 1 --2- 172.21.15.107:0/3602664391 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f65300668d0 0x7f6530068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:42.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.966+0000 7f65477fe640 1 -- 172.21.15.107:0/3602664391 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6538087400 con 0x7f6550158f60 2023-12-04T05:40:42.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.966+0000 7f654e59c640 1 --2- 172.21.15.107:0/3602664391 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f65300668d0 0x7f6530068d90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6540001750 tx=0x7f65400011a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:42.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:42.970+0000 7f65477fe640 1 -- 172.21.15.107:0/3602664391 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6550155120 con 0x7f6550158f60 2023-12-04T05:40:43.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.126+0000 7f6554895640 1 -- 172.21.15.107:0/3602664391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6550158da0 con 0x7f6550158f60 2023-12-04T05:40:43.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.130+0000 7f65477fe640 1 -- 172.21.15.107:0/3602664391 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16790 (secure 0 0 0) 0x7f6550158da0 con 0x7f6550158f60 2023-12-04T05:40:43.133 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:43.133 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":23,"fsid":"603e856b-31ac-475a-8f35-21ab7faadd48","created":"2023-12-04T05:39:34.921690+0000","modified":"2023-12-04T05:40:40.992888+0000","last_up_change":"2023-12-04T05:39:46.235790+0000","last_in_change":"2023-12-04T05:39:37.587639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T05:39:47.567702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T05:39:57.644286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T05:39:58.029897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ab7111c-0ae8-4d41-b6db-e3743614322c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6817","nonce":768714279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6821","nonce":768714279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6829","nonce":768714279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6825","nonce":768714279}]},"public_addr":"172.21.15.107:6817/768714279","cluster_addr":"172.21.15.107:6821/768714279","heartbeat_back_addr":"172.21.15.107:6829/768714279","heartbeat_front_addr":"172.21.15.107:6825/768714279","state":["exists","up"]},{"osd":1,"uuid":"baef7a1f-8026-4fab-9e70-48465271034e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6801","nonce":2496355981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6803","nonce":2496355981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6807","nonce":2496355981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6805","nonce":2496355981}]},"public_addr":"172.21.15.107:6801/2496355981","cluster_addr":"172.21.15.107:6803/2496355981","heartbeat_back_addr":"172.21.15.107:6807/2496355981","heartbeat_front_addr":"172.21.15.107:6805/2496355981","state":["exists","up"]},{"osd":2,"uuid":"90e8cf24-7b54-484a-9d85-b44a9d3dfeab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6809","nonce":3097298224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6811","nonce":3097298224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6815","nonce":3097298224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6813","nonce":3097298224}]},"public_addr":"172.21.15.107:6809/3097298224","cluster_addr":"172.21.15.107:6811/3097298224","heartbeat_back_addr":"172.21.15.107:6815/3097298224","heartbeat_front_addr":"172.21.15.107:6813/3097298224","state":["exists","up"]},{"osd":3,"uuid":"54658731-2b9a-4d5e-ab13-7c1be7d34e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6820","nonce":3773076671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6824","nonce":3773076671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6831","nonce":3773076671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6828","nonce":3773076671}]},"public_addr":"172.21.15.107:6820/3773076671","cluster_addr":"172.21.15.107:6824/3773076671","heartbeat_back_addr":"172.21.15.107:6831/3773076671","heartbeat_front_addr":"172.21.15.107:6828/3773076671","state":["exists","up"]},{"osd":4,"uuid":"9a9dea13-8afe-485d-aa43-45eda691179b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6801","nonce":2747823542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6803","nonce":2747823542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6807","nonce":2747823542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6805","nonce":2747823542}]},"public_addr":"172.21.15.195:6801/2747823542","cluster_addr":"172.21.15.195:6803/2747823542","heartbeat_back_addr":"172.21.15.195:6807/2747823542","heartbeat_front_addr":"172.21.15.195:6805/2747823542","state":["exists","up"]},{"osd":5,"uuid":"db840d83-5a7d-468a-8aa2-1997945847c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6825","nonce":1484264540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6827","nonce":1484264540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6831","nonce":1484264540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6829","nonce":1484264540}]},"public_addr":"172.21.15.195:6825/1484264540","cluster_addr":"172.21.15.195:6827/1484264540","heartbeat_back_addr":"172.21.15.195:6831/1484264540","heartbeat_front_addr":"172.21.15.195:6829/1484264540","state":["exists","up"]},{"osd":6,"uuid":"38b5f165-bf91-467e-a615-4b4784337689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6811","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6813","nonce":2638471412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6817","nonce":2638471412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6823","nonce":2638471412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6821","nonce":2638471412}]},"public_addr":"172.21.15.195:6813/2638471412","cluster_addr":"172.21.15.195:6817/2638471412","heartbeat_back_addr":"172.21.15.195:6823/2638471412","heartbeat_front_addr":"172.21.15.195:6821/2638471412","state":["exists","up"]},{"osd":7,"uuid":"e5b3c596-494b-4df3-9437-bc2d52168ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6809","nonce":768189614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6812","nonce":768189614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6820","nonce":768189614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6816","nonce":768189614}]},"public_addr":"172.21.15.195:6809/768189614","cluster_addr":"172.21.15.195:6812/768189614","heartbeat_back_addr":"172.21.15.195:6820/768189614","heartbeat_front_addr":"172.21.15.195:6816/768189614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.209077+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.086601+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.221568+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.238524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.059477+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.148422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.079867+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.064916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T05:40:43.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.130+0000 7f6554895640 1 -- 172.21.15.107:0/3602664391 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f65300668d0 msgr2=0x7f6530068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:43.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.130+0000 7f6554895640 1 --2- 172.21.15.107:0/3602664391 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f65300668d0 0x7f6530068d90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6540001750 tx=0x7f65400011a0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.130+0000 7f6554895640 1 -- 172.21.15.107:0/3602664391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6550158f60 msgr2=0x7f65501f1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.130+0000 7f6554895640 1 --2- 172.21.15.107:0/3602664391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6550158f60 0x7f65501f1180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f655006f1e0 tx=0x7f6538011ca0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.130+0000 7f6554895640 1 -- 172.21.15.107:0/3602664391 shutdown_connections 2023-12-04T05:40:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.130+0000 7f6554895640 1 --2- 172.21.15.107:0/3602664391 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f65300668d0 0x7f6530068d90 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.130+0000 7f6554895640 1 --2- 172.21.15.107:0/3602664391 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f655006d300 0x7f655014b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.130+0000 7f6554895640 1 --2- 172.21.15.107:0/3602664391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6550158f60 0x7f65501f1180 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.130+0000 7f6554895640 1 --2- 172.21.15.107:0/3602664391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f653c003980 0x7f655014a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.130+0000 7f6554895640 1 -- 172.21.15.107:0/3602664391 >> 172.21.15.107:0/3602664391 conn(0x7f65500c7ff0 msgr2=0x7f6550156cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.130+0000 7f6554895640 1 -- 172.21.15.107:0/3602664391 shutdown_connections 2023-12-04T05:40:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.130+0000 7f6554895640 1 -- 172.21.15.107:0/3602664391 wait complete. 2023-12-04T05:40:43.151 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-04T05:39:47.567702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-04T05:39:57.644286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-04T05:39:58.029897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-04T05:40:43.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-04T05:40:43.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.262+0000 7f1f84bec640 1 Processor -- start 2023-12-04T05:40:43.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.262+0000 7f1f84bec640 1 -- start start 2023-12-04T05:40:43.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.262+0000 7f1f84bec640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1f800724f0 0x7f1f80158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:43.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.262+0000 7f1f84bec640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1f800c56f0 0x7f1f80140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:43.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.262+0000 7f1f84bec640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f80158da0 con 0x7f1f80072110 2023-12-04T05:40:43.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.262+0000 7f1f84bec640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f801583d0 con 0x7f1f800c56f0 2023-12-04T05:40:43.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.262+0000 7f1f84bec640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f800c6170 con 0x7f1f800724f0 2023-12-04T05:40:43.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.262+0000 7f1f7ed9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1f800c56f0 0x7f1f80140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:43.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.262+0000 7f1f7ed9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1f800c56f0 0x7f1f80140830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56364/0 (socket says 172.21.15.107:56364) 2023-12-04T05:40:43.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.262+0000 7f1f7ed9d640 1 -- 172.21.15.107:0/413396045 learned_addr learned my addr 172.21.15.107:0/413396045 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:43.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.262+0000 7f1f7ed9d640 1 -- 172.21.15.107:0/413396045 >> v2:172.21.15.195:3300/0 conn(0x7f1f800724f0 msgr2=0x7f1f80158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:43.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.262+0000 7f1f7ed9d640 1 --2- 172.21.15.107:0/413396045 >> v2:172.21.15.195:3300/0 conn(0x7f1f800724f0 0x7f1f80158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.262+0000 7f1f7ed9d640 1 -- 172.21.15.107:0/413396045 >> v1:172.21.15.195:6789/0 conn(0x7f1f80072110 legacy=0x7f1f800c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:43.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.262+0000 7f1f7ed9d640 1 -- 172.21.15.107:0/413396045 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f80157da0 con 0x7f1f800c56f0 2023-12-04T05:40:43.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f7ed9d640 1 --2- 172.21.15.107:0/413396045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f800c56f0 0x7f1f80140830 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1f8006b870 tx=0x7f1f7801f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfa06e503cd68aa5 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:43.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f7dd9b640 1 -- 172.21.15.107:0/413396045 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f780af400 con 0x7f1f800c56f0 2023-12-04T05:40:43.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f7dd9b640 1 -- 172.21.15.107:0/413396045 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f780ab050 con 0x7f1f800c56f0 2023-12-04T05:40:43.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f7dd9b640 1 -- 172.21.15.107:0/413396045 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f780af780 con 0x7f1f800c56f0 2023-12-04T05:40:43.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f84bec640 1 -- 172.21.15.107:0/413396045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f800c56f0 msgr2=0x7f1f80140830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:43.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f84bec640 1 --2- 172.21.15.107:0/413396045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f800c56f0 0x7f1f80140830 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1f8006b870 tx=0x7f1f7801f180 comp rx=0 tx=0).stop 2023-12-04T05:40:43.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f84bec640 1 -- 172.21.15.107:0/413396045 shutdown_connections 2023-12-04T05:40:43.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f84bec640 1 --2- 172.21.15.107:0/413396045 >> v2:172.21.15.195:3300/0 conn(0x7f1f800724f0 0x7f1f80158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f84bec640 1 --2- 172.21.15.107:0/413396045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f800c56f0 0x7f1f80140830 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f84bec640 1 -- 172.21.15.107:0/413396045 >> 172.21.15.107:0/413396045 conn(0x7f1f800c7ff0 msgr2=0x7f1f800c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:43.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f84bec640 1 -- 172.21.15.107:0/413396045 shutdown_connections 2023-12-04T05:40:43.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f84bec640 1 -- 172.21.15.107:0/413396045 wait complete. 2023-12-04T05:40:43.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f84bec640 1 Processor -- start 2023-12-04T05:40:43.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f84bec640 1 -- start start 2023-12-04T05:40:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f84bec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f80072110 0x7f1f80155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f84bec640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f800724f0 0x7f1f8015b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f84bec640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1f800c56f0 0x7f1f8015b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f84bec640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f80158f60 con 0x7f1f80072110 2023-12-04T05:40:43.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f84bec640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8015a590 con 0x7f1f800724f0 2023-12-04T05:40:43.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f84bec640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f80159190 con 0x7f1f800c56f0 2023-12-04T05:40:43.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f7e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f80072110 0x7f1f80155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:43.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f7e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f80072110 0x7f1f80155e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56374/0 (socket says 172.21.15.107:56374) 2023-12-04T05:40:43.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f7e59c640 1 -- 172.21.15.107:0/4195127068 learned_addr learned my addr 172.21.15.107:0/4195127068 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:43.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f7e59c640 1 -- 172.21.15.107:0/4195127068 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1f800c56f0 msgr2=0x7f1f8015b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:43.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f75d9b640 1 --2- 172.21.15.107:0/4195127068 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f800724f0 0x7f1f8015b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:43.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f7e59c640 1 --2- 172.21.15.107:0/4195127068 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1f800c56f0 0x7f1f8015b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f7e59c640 1 -- 172.21.15.107:0/4195127068 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f800724f0 msgr2=0x7f1f8015b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:43.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f7e59c640 1 --2- 172.21.15.107:0/4195127068 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f800724f0 0x7f1f8015b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f7e59c640 1 -- 172.21.15.107:0/4195127068 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f78009b30 con 0x7f1f80072110 2023-12-04T05:40:43.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f75d9b640 1 --2- 172.21.15.107:0/4195127068 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f800724f0 0x7f1f8015b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:43.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f7e59c640 1 --2- 172.21.15.107:0/4195127068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f80072110 0x7f1f80155e70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1f7000c490 tx=0x7f1f70001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:43.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f777fe640 1 -- 172.21.15.107:0/4195127068 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f70003600 con 0x7f1f80072110 2023-12-04T05:40:43.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f777fe640 1 -- 172.21.15.107:0/4195127068 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f70003780 con 0x7f1f80072110 2023-12-04T05:40:43.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f84bec640 1 -- 172.21.15.107:0/4195127068 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f801ede10 con 0x7f1f80072110 2023-12-04T05:40:43.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f777fe640 1 -- 172.21.15.107:0/4195127068 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f70003b70 con 0x7f1f80072110 2023-12-04T05:40:43.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.266+0000 7f1f84bec640 1 -- 172.21.15.107:0/4195127068 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f801ee3d0 con 0x7f1f80072110 2023-12-04T05:40:43.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.270+0000 7f1f84bec640 1 -- 172.21.15.107:0/4195127068 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f80067870 con 0x7f1f80072110 2023-12-04T05:40:43.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.270+0000 7f1f777fe640 1 -- 172.21.15.107:0/4195127068 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f1f70003cd0 con 0x7f1f80072110 2023-12-04T05:40:43.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.270+0000 7f1f777fe640 1 --2- 172.21.15.107:0/4195127068 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1f60066900 0x7f1f60068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:43.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.270+0000 7f1f777fe640 1 -- 172.21.15.107:0/4195127068 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1f70083ad0 con 0x7f1f80072110 2023-12-04T05:40:43.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.270+0000 7f1f75d9b640 1 --2- 172.21.15.107:0/4195127068 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1f60066900 0x7f1f60068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:43.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.270+0000 7f1f75d9b640 1 --2- 172.21.15.107:0/4195127068 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1f60066900 0x7f1f60068dc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1f80159f10 tx=0x7f1f68008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:43.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.274+0000 7f1f777fe640 1 -- 172.21.15.107:0/4195127068 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1f70050c90 con 0x7f1f80072110 2023-12-04T05:40:43.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.418+0000 7f1f84bec640 1 -- 172.21.15.107:0/4195127068 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1f801570c0 con 0x7f1f80072110 2023-12-04T05:40:43.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.422+0000 7f1f777fe640 1 -- 172.21.15.107:0/4195127068 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1f70017070 con 0x7f1f80072110 2023-12-04T05:40:43.424 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-12-04T05:40:43.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.422+0000 7f1f84bec640 1 -- 172.21.15.107:0/4195127068 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1f60066900 msgr2=0x7f1f60068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:43.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.422+0000 7f1f84bec640 1 --2- 172.21.15.107:0/4195127068 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1f60066900 0x7f1f60068dc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1f80159f10 tx=0x7f1f68008040 comp rx=0 tx=0).stop 2023-12-04T05:40:43.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.422+0000 7f1f84bec640 1 -- 172.21.15.107:0/4195127068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f80072110 msgr2=0x7f1f80155e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:43.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.422+0000 7f1f84bec640 1 --2- 172.21.15.107:0/4195127068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f80072110 0x7f1f80155e70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1f7000c490 tx=0x7f1f70001e80 comp rx=0 tx=0).stop 2023-12-04T05:40:43.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.422+0000 7f1f84bec640 1 -- 172.21.15.107:0/4195127068 shutdown_connections 2023-12-04T05:40:43.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.422+0000 7f1f84bec640 1 --2- 172.21.15.107:0/4195127068 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1f60066900 0x7f1f60068dc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.422+0000 7f1f84bec640 1 --2- 172.21.15.107:0/4195127068 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1f800c56f0 0x7f1f8015b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.422+0000 7f1f84bec640 1 --2- 172.21.15.107:0/4195127068 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1f800724f0 0x7f1f8015b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.422+0000 7f1f84bec640 1 --2- 172.21.15.107:0/4195127068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f80072110 0x7f1f80155e70 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.422+0000 7f1f84bec640 1 -- 172.21.15.107:0/4195127068 >> 172.21.15.107:0/4195127068 conn(0x7f1f800c7ff0 msgr2=0x7f1f800c71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:43.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.422+0000 7f1f84bec640 1 -- 172.21.15.107:0/4195127068 shutdown_connections 2023-12-04T05:40:43.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.422+0000 7f1f84bec640 1 -- 172.21.15.107:0/4195127068 wait complete. 2023-12-04T05:40:43.441 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-04T05:40:43.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.558+0000 7f647e228640 1 Processor -- start 2023-12-04T05:40:43.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.558+0000 7f647e228640 1 -- start start 2023-12-04T05:40:43.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.558+0000 7f647e228640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6478072570 0x7f6478072980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:43.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.558+0000 7f647e228640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f64780ca010 0x7f64780ca430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:43.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.558+0000 7f647e228640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6478159ee0 con 0x7f6478150a20 2023-12-04T05:40:43.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.558+0000 7f647e228640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6478150e00 con 0x7f64780ca010 2023-12-04T05:40:43.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.558+0000 7f647e228640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647815a140 con 0x7f6478072570 2023-12-04T05:40:43.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.558+0000 7f6477fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f64780ca010 0x7f64780ca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:43.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.558+0000 7f6477fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f64780ca010 0x7f64780ca430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56388/0 (socket says 172.21.15.107:56388) 2023-12-04T05:40:43.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.558+0000 7f6477fff640 1 -- 172.21.15.107:0/1154183705 learned_addr learned my addr 172.21.15.107:0/1154183705 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:43.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.558+0000 7f6476ffd640 1 --2- 172.21.15.107:0/1154183705 >> v2:172.21.15.195:3300/0 conn(0x7f6478072570 0x7f6478072980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:43.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.558+0000 7f6477fff640 1 -- 172.21.15.107:0/1154183705 >> v2:172.21.15.195:3300/0 conn(0x7f6478072570 msgr2=0x7f6478072980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:43.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.558+0000 7f6477fff640 1 --2- 172.21.15.107:0/1154183705 >> v2:172.21.15.195:3300/0 conn(0x7f6478072570 0x7f6478072980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.558+0000 7f6477fff640 1 -- 172.21.15.107:0/1154183705 >> v1:172.21.15.195:6789/0 conn(0x7f6478150a20 legacy=0x7f647806ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:43.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.558+0000 7f6477fff640 1 -- 172.21.15.107:0/1154183705 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647815a420 con 0x7f64780ca010 2023-12-04T05:40:43.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.558+0000 7f6476ffd640 1 --2- 172.21.15.107:0/1154183705 >> v2:172.21.15.195:3300/0 conn(0x7f6478072570 0x7f6478072980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:43.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f6477fff640 1 --2- 172.21.15.107:0/1154183705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64780ca010 0x7f64780ca430 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6478067350 tx=0x7f647001f160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3a66feeb9690e8 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:43.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f64767fc640 1 -- 172.21.15.107:0/1154183705 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64700af400 con 0x7f64780ca010 2023-12-04T05:40:43.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f64767fc640 1 -- 172.21.15.107:0/1154183705 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64700ab050 con 0x7f64780ca010 2023-12-04T05:40:43.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f64767fc640 1 -- 172.21.15.107:0/1154183705 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64700af780 con 0x7f64780ca010 2023-12-04T05:40:43.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f647e228640 1 -- 172.21.15.107:0/1154183705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64780ca010 msgr2=0x7f64780ca430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:43.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f647e228640 1 --2- 172.21.15.107:0/1154183705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64780ca010 0x7f64780ca430 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6478067350 tx=0x7f647001f160 comp rx=0 tx=0).stop 2023-12-04T05:40:43.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f647e228640 1 -- 172.21.15.107:0/1154183705 shutdown_connections 2023-12-04T05:40:43.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f647e228640 1 --2- 172.21.15.107:0/1154183705 >> v2:172.21.15.195:3300/0 conn(0x7f6478072570 0x7f6478072980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f647e228640 1 --2- 172.21.15.107:0/1154183705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64780ca010 0x7f64780ca430 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f647e228640 1 -- 172.21.15.107:0/1154183705 >> 172.21.15.107:0/1154183705 conn(0x7f64780663a0 msgr2=0x7f64780667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:43.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f647e228640 1 -- 172.21.15.107:0/1154183705 shutdown_connections 2023-12-04T05:40:43.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f647e228640 1 -- 172.21.15.107:0/1154183705 wait complete. 2023-12-04T05:40:43.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f647e228640 1 Processor -- start 2023-12-04T05:40:43.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f647e228640 1 -- start start 2023-12-04T05:40:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f647e228640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6478072570 0x7f64780c80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f647e228640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64780ca010 0x7f64781f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f647e228640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6478150a20 0x7f64780c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f647e228640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64781f38b0 con 0x7f6478072570 2023-12-04T05:40:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f647e228640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6478149620 con 0x7f64780ca010 2023-12-04T05:40:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f647e228640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64781f3b50 con 0x7f6478150a20 2023-12-04T05:40:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f64777fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6478072570 0x7f64780c80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f64777fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6478072570 0x7f64780c80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56404/0 (socket says 172.21.15.107:56404) 2023-12-04T05:40:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f64777fe640 1 -- 172.21.15.107:0/3148024185 learned_addr learned my addr 172.21.15.107:0/3148024185 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:43.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f64777fe640 1 -- 172.21.15.107:0/3148024185 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6478150a20 msgr2=0x7f64780c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:43.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f64777fe640 1 --2- 172.21.15.107:0/3148024185 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6478150a20 0x7f64780c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f64777fe640 1 -- 172.21.15.107:0/3148024185 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64780ca010 msgr2=0x7f64781f2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:43.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f64777fe640 1 --2- 172.21.15.107:0/3148024185 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64780ca010 0x7f64781f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f64777fe640 1 -- 172.21.15.107:0/3148024185 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6470009b30 con 0x7f6478072570 2023-12-04T05:40:43.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f64777fe640 1 --2- 172.21.15.107:0/3148024185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6478072570 0x7f64780c80d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f646c00b410 tx=0x7f646c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:43.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f645ffff640 1 -- 172.21.15.107:0/3148024185 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c004020 con 0x7f6478072570 2023-12-04T05:40:43.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f645ffff640 1 -- 172.21.15.107:0/3148024185 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646c004180 con 0x7f6478072570 2023-12-04T05:40:43.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f647e228640 1 -- 172.21.15.107:0/3148024185 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64781498c0 con 0x7f6478072570 2023-12-04T05:40:43.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f645ffff640 1 -- 172.21.15.107:0/3148024185 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c011630 con 0x7f6478072570 2023-12-04T05:40:43.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.562+0000 7f647e228640 1 -- 172.21.15.107:0/3148024185 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64781f3520 con 0x7f6478072570 2023-12-04T05:40:43.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.566+0000 7f647e228640 1 -- 172.21.15.107:0/3148024185 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6478149b00 con 0x7f6478072570 2023-12-04T05:40:43.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.566+0000 7f645ffff640 1 -- 172.21.15.107:0/3148024185 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f646c011790 con 0x7f6478072570 2023-12-04T05:40:43.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.566+0000 7f645ffff640 1 --2- 172.21.15.107:0/3148024185 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6464066900 0x7f6464068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:43.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.566+0000 7f645ffff640 1 -- 172.21.15.107:0/3148024185 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f646c084290 con 0x7f6478072570 2023-12-04T05:40:43.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.566+0000 7f6476ffd640 1 --2- 172.21.15.107:0/3148024185 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6464066900 0x7f6464068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:43.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.566+0000 7f6476ffd640 1 --2- 172.21.15.107:0/3148024185 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6464066900 0x7f6464068dc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f64780c8610 tx=0x7f6468008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:43.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.570+0000 7f645ffff640 1 -- 172.21.15.107:0/3148024185 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6478149b00 con 0x7f6478072570 2023-12-04T05:40:43.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.710+0000 7f647e228640 1 -- 172.21.15.107:0/3148024185 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6478149b00 con 0x7f6478072570 2023-12-04T05:40:43.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.710+0000 7f645ffff640 1 -- 172.21.15.107:0/3148024185 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6478149b00 con 0x7f6478072570 2023-12-04T05:40:43.714 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-12-04T05:40:43.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.710+0000 7f647e228640 1 -- 172.21.15.107:0/3148024185 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6464066900 msgr2=0x7f6464068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:43.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.710+0000 7f647e228640 1 --2- 172.21.15.107:0/3148024185 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6464066900 0x7f6464068dc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f64780c8610 tx=0x7f6468008040 comp rx=0 tx=0).stop 2023-12-04T05:40:43.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.710+0000 7f647e228640 1 -- 172.21.15.107:0/3148024185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6478072570 msgr2=0x7f64780c80d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:43.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.710+0000 7f647e228640 1 --2- 172.21.15.107:0/3148024185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6478072570 0x7f64780c80d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f646c00b410 tx=0x7f646c00b8d0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.710+0000 7f647e228640 1 -- 172.21.15.107:0/3148024185 shutdown_connections 2023-12-04T05:40:43.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.710+0000 7f647e228640 1 --2- 172.21.15.107:0/3148024185 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6464066900 0x7f6464068dc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.710+0000 7f647e228640 1 --2- 172.21.15.107:0/3148024185 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6478150a20 0x7f64780c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.710+0000 7f647e228640 1 --2- 172.21.15.107:0/3148024185 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64780ca010 0x7f64781f2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.710+0000 7f647e228640 1 --2- 172.21.15.107:0/3148024185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6478072570 0x7f64780c80d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.710+0000 7f647e228640 1 -- 172.21.15.107:0/3148024185 >> 172.21.15.107:0/3148024185 conn(0x7f64780663a0 msgr2=0x7f6478066780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:43.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.710+0000 7f647e228640 1 -- 172.21.15.107:0/3148024185 shutdown_connections 2023-12-04T05:40:43.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.710+0000 7f647e228640 1 -- 172.21.15.107:0/3148024185 wait complete. 2023-12-04T05:40:43.731 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-04T05:40:43.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.842+0000 7f7018a30640 1 Processor -- start 2023-12-04T05:40:43.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.846+0000 7f7018a30640 1 -- start start 2023-12-04T05:40:43.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.846+0000 7f7018a30640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f70140724f0 0x7f7014158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:43.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.846+0000 7f7018a30640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70140c56f0 0x7f7014140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:43.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.846+0000 7f7018a30640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7014158da0 con 0x7f7014072110 2023-12-04T05:40:43.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.846+0000 7f7018a30640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70141583d0 con 0x7f70140724f0 2023-12-04T05:40:43.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.846+0000 7f7018a30640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70140c6170 con 0x7f70140c56f0 2023-12-04T05:40:43.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.846+0000 7f7012d9d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70140c56f0 0x7f7014140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:43.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.846+0000 7f7009d9b640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f70140724f0 0x7f7014158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:43.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.846+0000 7f7012d9d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70140c56f0 0x7f7014140830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.107:60826/0 (socket says 172.21.15.107:60826) 2023-12-04T05:40:43.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.846+0000 7f701259c640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f7014072110 0x7f70140c7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:50646/0 (socket says 172.21.15.107:50646) 2023-12-04T05:40:43.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.846+0000 7f7012d9d640 1 -- 172.21.15.107:0/240124201 learned_addr learned my addr 172.21.15.107:0/240124201 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:43.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.846+0000 7f7012d9d640 1 -- 172.21.15.107:0/240124201 >> v2:172.21.15.195:3300/0 conn(0x7f70140724f0 msgr2=0x7f7014158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:43.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.846+0000 7f7012d9d640 1 --2- 172.21.15.107:0/240124201 >> v2:172.21.15.195:3300/0 conn(0x7f70140724f0 0x7f7014158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.846+0000 7f7012d9d640 1 -- 172.21.15.107:0/240124201 >> v1:172.21.15.195:6789/0 conn(0x7f7014072110 legacy=0x7f70140c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:43.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.846+0000 7f7012d9d640 1 -- 172.21.15.107:0/240124201 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7014157da0 con 0x7f70140c56f0 2023-12-04T05:40:43.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7012d9d640 1 --2- 172.21.15.107:0/240124201 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70140c56f0 0x7f7014140830 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f700c009e50 tx=0x7f700c01ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22927b5df3792849 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:43.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7011d9b640 1 -- 172.21.15.107:0/240124201 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c0a57d0 con 0x7f70140c56f0 2023-12-04T05:40:43.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7011d9b640 1 -- 172.21.15.107:0/240124201 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700c0a5930 con 0x7f70140c56f0 2023-12-04T05:40:43.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7011d9b640 1 -- 172.21.15.107:0/240124201 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c0a5cc0 con 0x7f70140c56f0 2023-12-04T05:40:43.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7018a30640 1 -- 172.21.15.107:0/240124201 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70140c56f0 msgr2=0x7f7014140830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:43.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7018a30640 1 --2- 172.21.15.107:0/240124201 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70140c56f0 0x7f7014140830 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f700c009e50 tx=0x7f700c01ca10 comp rx=0 tx=0).stop 2023-12-04T05:40:43.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7018a30640 1 -- 172.21.15.107:0/240124201 shutdown_connections 2023-12-04T05:40:43.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7018a30640 1 --2- 172.21.15.107:0/240124201 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70140c56f0 0x7f7014140830 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7018a30640 1 --2- 172.21.15.107:0/240124201 >> v2:172.21.15.195:3300/0 conn(0x7f70140724f0 0x7f7014158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7018a30640 1 -- 172.21.15.107:0/240124201 >> 172.21.15.107:0/240124201 conn(0x7f70140c7ff0 msgr2=0x7f70140c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:43.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7018a30640 1 -- 172.21.15.107:0/240124201 shutdown_connections 2023-12-04T05:40:43.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7018a30640 1 -- 172.21.15.107:0/240124201 wait complete. 2023-12-04T05:40:43.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7018a30640 1 Processor -- start 2023-12-04T05:40:43.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7018a30640 1 -- start start 2023-12-04T05:40:43.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7018a30640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7014072110 0x7f70141626f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:43.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7018a30640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70140724f0 0x7f701406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:43.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7018a30640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70140c56f0 0x7f701413ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:43.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7018a30640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7014159da0 con 0x7f70140c56f0 2023-12-04T05:40:43.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7018a30640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70141f70b0 con 0x7f7014072110 2023-12-04T05:40:43.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7018a30640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70141f64a0 con 0x7f70140724f0 2023-12-04T05:40:43.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7012d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70140c56f0 0x7f701413ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:43.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7012d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70140c56f0 0x7f701413ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56412/0 (socket says 172.21.15.107:56412) 2023-12-04T05:40:43.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7012d9d640 1 -- 172.21.15.107:0/1615378594 learned_addr learned my addr 172.21.15.107:0/1615378594 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:43.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7009d9b640 1 --2- 172.21.15.107:0/1615378594 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70140724f0 0x7f701406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:43.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f701259c640 1 --2- 172.21.15.107:0/1615378594 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7014072110 0x7f70141626f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:43.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7012d9d640 1 -- 172.21.15.107:0/1615378594 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70140724f0 msgr2=0x7f701406d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:43.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7012d9d640 1 --2- 172.21.15.107:0/1615378594 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70140724f0 0x7f701406d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7012d9d640 1 -- 172.21.15.107:0/1615378594 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7014072110 msgr2=0x7f70141626f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:43.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7012d9d640 1 --2- 172.21.15.107:0/1615378594 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7014072110 0x7f70141626f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:43.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.850+0000 7f7012d9d640 1 -- 172.21.15.107:0/1615378594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700c009b30 con 0x7f70140c56f0 2023-12-04T05:40:43.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.854+0000 7f7012d9d640 1 --2- 172.21.15.107:0/1615378594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70140c56f0 0x7f701413ce70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f700c01a640 tx=0x7f700c0197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:43.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.854+0000 7f700b7fe640 1 -- 172.21.15.107:0/1615378594 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c0b8670 con 0x7f70140c56f0 2023-12-04T05:40:43.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.854+0000 7f700b7fe640 1 -- 172.21.15.107:0/1615378594 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700c0b87f0 con 0x7f70140c56f0 2023-12-04T05:40:43.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.854+0000 7f700b7fe640 1 -- 172.21.15.107:0/1615378594 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c0b8ba0 con 0x7f70140c56f0 2023-12-04T05:40:43.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.854+0000 7f7018a30640 1 -- 172.21.15.107:0/1615378594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70141f6780 con 0x7f70140c56f0 2023-12-04T05:40:43.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.854+0000 7f7018a30640 1 -- 172.21.15.107:0/1615378594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70141f6d40 con 0x7f70140c56f0 2023-12-04T05:40:43.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.854+0000 7f7018a30640 1 -- 172.21.15.107:0/1615378594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70141f7220 con 0x7f70140c56f0 2023-12-04T05:40:43.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.854+0000 7f700b7fe640 1 -- 172.21.15.107:0/1615378594 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f700c0b8d00 con 0x7f70140c56f0 2023-12-04T05:40:43.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.854+0000 7f700b7fe640 1 --2- 172.21.15.107:0/1615378594 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6ff4066900 0x7f6ff4068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:43.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.854+0000 7f700b7fe640 1 -- 172.21.15.107:0/1615378594 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f700c11f940 con 0x7f70140c56f0 2023-12-04T05:40:43.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.854+0000 7f701259c640 1 --2- 172.21.15.107:0/1615378594 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6ff4066900 0x7f6ff4068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:43.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.854+0000 7f701259c640 1 --2- 172.21.15.107:0/1615378594 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6ff4066900 0x7f6ff4068dc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7004000f70 tx=0x7f7004007c50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:43.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:43.858+0000 7f700b7fe640 1 -- 172.21.15.107:0/1615378594 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f700c0ecc20 con 0x7f70140c56f0 2023-12-04T05:40:44.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.002+0000 7f7018a30640 1 -- 172.21.15.107:0/1615378594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7014067870 con 0x7f70140c56f0 2023-12-04T05:40:44.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.002+0000 7f700b7fe640 1 -- 172.21.15.107:0/1615378594 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f700c0f0240 con 0x7f70140c56f0 2023-12-04T05:40:44.006 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-12-04T05:40:44.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.002+0000 7f7018a30640 1 -- 172.21.15.107:0/1615378594 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6ff4066900 msgr2=0x7f6ff4068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:44.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.002+0000 7f7018a30640 1 --2- 172.21.15.107:0/1615378594 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6ff4066900 0x7f6ff4068dc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7004000f70 tx=0x7f7004007c50 comp rx=0 tx=0).stop 2023-12-04T05:40:44.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.002+0000 7f7018a30640 1 -- 172.21.15.107:0/1615378594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70140c56f0 msgr2=0x7f701413ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:44.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.002+0000 7f7018a30640 1 --2- 172.21.15.107:0/1615378594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70140c56f0 0x7f701413ce70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f700c01a640 tx=0x7f700c0197b0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.002+0000 7f7018a30640 1 -- 172.21.15.107:0/1615378594 shutdown_connections 2023-12-04T05:40:44.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.002+0000 7f7018a30640 1 --2- 172.21.15.107:0/1615378594 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6ff4066900 0x7f6ff4068dc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.002+0000 7f7018a30640 1 --2- 172.21.15.107:0/1615378594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70140c56f0 0x7f701413ce70 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.002+0000 7f7018a30640 1 --2- 172.21.15.107:0/1615378594 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70140724f0 0x7f701406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.002+0000 7f7018a30640 1 --2- 172.21.15.107:0/1615378594 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7014072110 0x7f70141626f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.002+0000 7f7018a30640 1 -- 172.21.15.107:0/1615378594 >> 172.21.15.107:0/1615378594 conn(0x7f70140c7ff0 msgr2=0x7f70140c92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:44.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.002+0000 7f7018a30640 1 -- 172.21.15.107:0/1615378594 shutdown_connections 2023-12-04T05:40:44.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.006+0000 7f7018a30640 1 -- 172.21.15.107:0/1615378594 wait complete. 2023-12-04T05:40:44.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:40:44.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.122+0000 7f302fa86640 1 Processor -- start 2023-12-04T05:40:44.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.122+0000 7f302fa86640 1 -- start start 2023-12-04T05:40:44.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302fa86640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3028151180 0x7f30280c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:44.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302fa86640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f30280714f0 0x7f30280718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:44.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302fa86640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3028070380 0x7f3028070790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:44.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302fa86640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30280700c0 con 0x7f30280714f0 2023-12-04T05:40:44.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302fa86640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3028156df0 con 0x7f3028070380 2023-12-04T05:40:44.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302fa86640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30281570d0 con 0x7f3028151180 2023-12-04T05:40:44.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302d021640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f30280714f0 0x7f30280718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:44.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302d021640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f30280714f0 0x7f30280718f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56418/0 (socket says 172.21.15.107:56418) 2023-12-04T05:40:44.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302d021640 1 -- 172.21.15.107:0/1129220320 learned_addr learned my addr 172.21.15.107:0/1129220320 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:44.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302d021640 1 -- 172.21.15.107:0/1129220320 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3028151180 msgr2=0x7f30280c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:44.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302d021640 1 --2- 172.21.15.107:0/1129220320 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3028151180 0x7f30280c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302d021640 1 -- 172.21.15.107:0/1129220320 >> v2:172.21.15.195:3300/0 conn(0x7f3028070380 msgr2=0x7f3028070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:44.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302d021640 1 --2- 172.21.15.107:0/1129220320 >> v2:172.21.15.195:3300/0 conn(0x7f3028070380 0x7f3028070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302d021640 1 -- 172.21.15.107:0/1129220320 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30281573b0 con 0x7f30280714f0 2023-12-04T05:40:44.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302d021640 1 --2- 172.21.15.107:0/1129220320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30280714f0 0x7f30280718f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f302806e5e0 tx=0x7f301801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=812b484e7c88761 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:44.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302c820640 1 -- 172.21.15.107:0/1129220320 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30180a89b0 con 0x7f30280714f0 2023-12-04T05:40:44.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302c820640 1 -- 172.21.15.107:0/1129220320 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30180a8b10 con 0x7f30280714f0 2023-12-04T05:40:44.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302c820640 1 -- 172.21.15.107:0/1129220320 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30180a8ea0 con 0x7f30280714f0 2023-12-04T05:40:44.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302fa86640 1 -- 172.21.15.107:0/1129220320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30280714f0 msgr2=0x7f30280718f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:44.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302fa86640 1 --2- 172.21.15.107:0/1129220320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30280714f0 0x7f30280718f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f302806e5e0 tx=0x7f301801ecf0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302fa86640 1 -- 172.21.15.107:0/1129220320 shutdown_connections 2023-12-04T05:40:44.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302fa86640 1 --2- 172.21.15.107:0/1129220320 >> v2:172.21.15.195:3300/0 conn(0x7f3028070380 0x7f3028070790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302fa86640 1 --2- 172.21.15.107:0/1129220320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30280714f0 0x7f30280718f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302fa86640 1 --2- 172.21.15.107:0/1129220320 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3028151180 0x7f30280c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302fa86640 1 -- 172.21.15.107:0/1129220320 >> 172.21.15.107:0/1129220320 conn(0x7f30280c7ff0 msgr2=0x7f30280c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:44.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302fa86640 1 -- 172.21.15.107:0/1129220320 shutdown_connections 2023-12-04T05:40:44.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.126+0000 7f302fa86640 1 -- 172.21.15.107:0/1129220320 wait complete. 2023-12-04T05:40:44.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302fa86640 1 Processor -- start 2023-12-04T05:40:44.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302fa86640 1 -- start start 2023-12-04T05:40:44.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302fa86640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3028070380 0x7f3028150140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302fa86640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30280714f0 0x7f302813c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302fa86640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3028151180 0x7f30281edf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302fa86640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302813e190 con 0x7f30280714f0 2023-12-04T05:40:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302fa86640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302813dbd0 con 0x7f3028151180 2023-12-04T05:40:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302fa86640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f302813deb0 con 0x7f3028070380 2023-12-04T05:40:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302d021640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30280714f0 0x7f302813c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302d021640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30280714f0 0x7f302813c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56426/0 (socket says 172.21.15.107:56426) 2023-12-04T05:40:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302d021640 1 -- 172.21.15.107:0/834724083 learned_addr learned my addr 172.21.15.107:0/834724083 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302d822640 1 --2- 172.21.15.107:0/834724083 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3028070380 0x7f3028150140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302d021640 1 -- 172.21.15.107:0/834724083 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3028070380 msgr2=0x7f3028150140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302e023640 1 --2- 172.21.15.107:0/834724083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3028151180 0x7f30281edf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302d021640 1 --2- 172.21.15.107:0/834724083 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3028070380 0x7f3028150140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302d021640 1 -- 172.21.15.107:0/834724083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3028151180 msgr2=0x7f30281edf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302d021640 1 --2- 172.21.15.107:0/834724083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3028151180 0x7f30281edf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302d021640 1 -- 172.21.15.107:0/834724083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30180094d0 con 0x7f30280714f0 2023-12-04T05:40:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302d822640 1 --2- 172.21.15.107:0/834724083 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3028070380 0x7f3028150140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:40:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302d021640 1 --2- 172.21.15.107:0/834724083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30280714f0 0x7f302813c640 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3018000c10 tx=0x7f301801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f301e7fc640 1 -- 172.21.15.107:0/834724083 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30180c1070 con 0x7f30280714f0 2023-12-04T05:40:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f301e7fc640 1 -- 172.21.15.107:0/834724083 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30180b0c80 con 0x7f30280714f0 2023-12-04T05:40:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302fa86640 1 -- 172.21.15.107:0/834724083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f302813d1a0 con 0x7f30280714f0 2023-12-04T05:40:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f301e7fc640 1 -- 172.21.15.107:0/834724083 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30180bb420 con 0x7f30280714f0 2023-12-04T05:40:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302fa86640 1 -- 172.21.15.107:0/834724083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30280c74b0 con 0x7f30280714f0 2023-12-04T05:40:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.130+0000 7f302fa86640 1 -- 172.21.15.107:0/834724083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f302813d000 con 0x7f30280714f0 2023-12-04T05:40:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.134+0000 7f301e7fc640 1 -- 172.21.15.107:0/834724083 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f30180b7040 con 0x7f30280714f0 2023-12-04T05:40:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.134+0000 7f301e7fc640 1 --2- 172.21.15.107:0/834724083 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f300c0668d0 0x7f300c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.134+0000 7f301e7fc640 1 -- 172.21.15.107:0/834724083 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3018121eb0 con 0x7f30280714f0 2023-12-04T05:40:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.134+0000 7f302d822640 1 --2- 172.21.15.107:0/834724083 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f300c0668d0 0x7f300c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.134+0000 7f302d822640 1 --2- 172.21.15.107:0/834724083 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f300c0668d0 0x7f300c068d90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f302400a870 tx=0x7f3024008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:44.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.138+0000 7f301e7fc640 1 -- 172.21.15.107:0/834724083 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f302813d000 con 0x7f30280714f0 2023-12-04T05:40:44.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.310+0000 7f302fa86640 1 -- 172.21.15.107:0/834724083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f302813d000 con 0x7f30280714f0 2023-12-04T05:40:44.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.310+0000 7f301e7fc640 1 -- 172.21.15.107:0/834724083 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6563 (secure 0 0 0) 0x7f302813d000 con 0x7f30280714f0 2023-12-04T05:40:44.314 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:44.314 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:40:28.948661+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":10,"addr":"172.21.15.195:6836/3769196693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":3769196693},{"type":"v1","addr":"172.21.15.195:6836","nonce":3769196693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:40:44.314 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-12-04T05:40:44.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.310+0000 7f302fa86640 1 -- 172.21.15.107:0/834724083 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f300c0668d0 msgr2=0x7f300c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:44.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.310+0000 7f302fa86640 1 --2- 172.21.15.107:0/834724083 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f300c0668d0 0x7f300c068d90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f302400a870 tx=0x7f3024008040 comp rx=0 tx=0).stop 2023-12-04T05:40:44.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.310+0000 7f302fa86640 1 -- 172.21.15.107:0/834724083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30280714f0 msgr2=0x7f302813c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:44.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.310+0000 7f302fa86640 1 --2- 172.21.15.107:0/834724083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30280714f0 0x7f302813c640 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3018000c10 tx=0x7f301801ba90 comp rx=0 tx=0).stop 2023-12-04T05:40:44.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.310+0000 7f302fa86640 1 -- 172.21.15.107:0/834724083 shutdown_connections 2023-12-04T05:40:44.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.310+0000 7f302fa86640 1 --2- 172.21.15.107:0/834724083 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f300c0668d0 0x7f300c068d90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.310+0000 7f302fa86640 1 --2- 172.21.15.107:0/834724083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3028151180 0x7f30281edf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.310+0000 7f302fa86640 1 --2- 172.21.15.107:0/834724083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30280714f0 0x7f302813c640 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.310+0000 7f302fa86640 1 --2- 172.21.15.107:0/834724083 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3028070380 0x7f3028150140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.310+0000 7f302fa86640 1 -- 172.21.15.107:0/834724083 >> 172.21.15.107:0/834724083 conn(0x7f30280c7ff0 msgr2=0x7f3028070760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:44.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.310+0000 7f302fa86640 1 -- 172.21.15.107:0/834724083 shutdown_connections 2023-12-04T05:40:44.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.310+0000 7f302fa86640 1 -- 172.21.15.107:0/834724083 wait complete. 2023-12-04T05:40:44.331 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T05:40:44.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.446+0000 7f853ebe6640 1 Processor -- start 2023-12-04T05:40:44.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.446+0000 7f853ebe6640 1 -- start start 2023-12-04T05:40:44.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.446+0000 7f853ebe6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f85380724f0 0x7f8538158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:44.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.446+0000 7f853ebe6640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f85380c56f0 0x7f8538140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:44.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.446+0000 7f853ebe6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8538158da0 con 0x7f8538072110 2023-12-04T05:40:44.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.446+0000 7f853ebe6640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85381583d0 con 0x7f85380724f0 2023-12-04T05:40:44.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.446+0000 7f853ebe6640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85380c6170 con 0x7f85380c56f0 2023-12-04T05:40:44.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.446+0000 7f8537fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f85380724f0 0x7f8538158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:44.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.446+0000 7f853c982640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8538072110 0x7f85380c7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39644/0 (socket says 172.21.15.107:39644) 2023-12-04T05:40:44.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.446+0000 7f853c982640 1 -- 172.21.15.107:0/875150464 learned_addr learned my addr 172.21.15.107:0/875150464 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:40:44.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.446+0000 7f853d183640 1 --2- 172.21.15.107:0/875150464 >> v2:172.21.15.195:3300/0 conn(0x7f85380c56f0 0x7f8538140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:44.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.446+0000 7f8537fff640 1 -- 172.21.15.107:0/875150464 >> v2:172.21.15.195:3300/0 conn(0x7f85380c56f0 msgr2=0x7f8538140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:40:44.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.446+0000 7f8537fff640 1 --2- 172.21.15.107:0/875150464 >> v2:172.21.15.195:3300/0 conn(0x7f85380c56f0 0x7f8538140830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.446+0000 7f8537fff640 1 -- 172.21.15.107:0/875150464 >> v1:172.21.15.107:6789/0 conn(0x7f8538072110 legacy=0x7f85380c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:44.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.446+0000 7f8537fff640 1 -- 172.21.15.107:0/875150464 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8538157da0 con 0x7f85380724f0 2023-12-04T05:40:44.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.446+0000 7f85377fe640 1 -- 172.21.15.107:0/875150464 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711046401 0 0) 0x7f8528001120 con 0x7f8538072110 2023-12-04T05:40:44.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f8537fff640 1 --2- 172.21.15.107:0/875150464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85380724f0 0x7f8538158860 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f85380c9080 tx=0x7f852001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ee5b1ae58540610 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:44.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f85377fe640 1 -- 172.21.15.107:0/875150464 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85200a8950 con 0x7f85380724f0 2023-12-04T05:40:44.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f85377fe640 1 -- 172.21.15.107:0/875150464 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85200a8ab0 con 0x7f85380724f0 2023-12-04T05:40:44.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f85377fe640 1 -- 172.21.15.107:0/875150464 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85200a8e40 con 0x7f85380724f0 2023-12-04T05:40:44.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f853ebe6640 1 -- 172.21.15.107:0/875150464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85380724f0 msgr2=0x7f8538158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:44.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f853ebe6640 1 --2- 172.21.15.107:0/875150464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85380724f0 0x7f8538158860 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f85380c9080 tx=0x7f852001ec50 comp rx=0 tx=0).stop 2023-12-04T05:40:44.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f853ebe6640 1 -- 172.21.15.107:0/875150464 shutdown_connections 2023-12-04T05:40:44.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f853ebe6640 1 --2- 172.21.15.107:0/875150464 >> v2:172.21.15.195:3300/0 conn(0x7f85380c56f0 0x7f8538140830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f853ebe6640 1 --2- 172.21.15.107:0/875150464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85380724f0 0x7f8538158860 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f853ebe6640 1 -- 172.21.15.107:0/875150464 >> 172.21.15.107:0/875150464 conn(0x7f85380c7ff0 msgr2=0x7f85380c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:44.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f853ebe6640 1 -- 172.21.15.107:0/875150464 shutdown_connections 2023-12-04T05:40:44.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f853ebe6640 1 -- 172.21.15.107:0/875150464 wait complete. 2023-12-04T05:40:44.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f853ebe6640 1 Processor -- start 2023-12-04T05:40:44.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f853ebe6640 1 -- start start 2023-12-04T05:40:44.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f853ebe6640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8538072110 0x7f85381f0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:44.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f853ebe6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85380724f0 0x7f85381528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:44.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f853ebe6640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f85380c56f0 0x7f8538152e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:44.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f853ebe6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8538153370 con 0x7f85380724f0 2023-12-04T05:40:44.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f853ebe6640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8538154580 con 0x7f8538072110 2023-12-04T05:40:44.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f853ebe6640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85381f2a60 con 0x7f85380c56f0 2023-12-04T05:40:44.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f8537fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85380724f0 0x7f85381528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:44.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f8537fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85380724f0 0x7f85381528f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56448/0 (socket says 172.21.15.107:56448) 2023-12-04T05:40:44.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f8537fff640 1 -- 172.21.15.107:0/915991455 learned_addr learned my addr 172.21.15.107:0/915991455 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:40:44.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f8537fff640 1 -- 172.21.15.107:0/915991455 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f85380c56f0 msgr2=0x7f8538152e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:44.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f8537fff640 1 --2- 172.21.15.107:0/915991455 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f85380c56f0 0x7f8538152e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f8537fff640 1 -- 172.21.15.107:0/915991455 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8538072110 msgr2=0x7f85381f0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:44.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f853c982640 1 --2- 172.21.15.107:0/915991455 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8538072110 0x7f85381f0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:44.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f8537fff640 1 --2- 172.21.15.107:0/915991455 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8538072110 0x7f85381f0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f8537fff640 1 -- 172.21.15.107:0/915991455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8520009420 con 0x7f85380724f0 2023-12-04T05:40:44.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f8537fff640 1 --2- 172.21.15.107:0/915991455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85380724f0 0x7f85381528f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f85200b3040 tx=0x7f852001b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:44.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.450+0000 7f85357fa640 1 -- 172.21.15.107:0/915991455 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85200b9630 con 0x7f85380724f0 2023-12-04T05:40:44.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.454+0000 7f85357fa640 1 -- 172.21.15.107:0/915991455 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85200b97b0 con 0x7f85380724f0 2023-12-04T05:40:44.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.454+0000 7f853ebe6640 1 -- 172.21.15.107:0/915991455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85381f2d60 con 0x7f85380724f0 2023-12-04T05:40:44.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.454+0000 7f85357fa640 1 -- 172.21.15.107:0/915991455 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85200b9b60 con 0x7f85380724f0 2023-12-04T05:40:44.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.454+0000 7f853ebe6640 1 -- 172.21.15.107:0/915991455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f853815b590 con 0x7f85380724f0 2023-12-04T05:40:44.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.454+0000 7f853ebe6640 1 -- 172.21.15.107:0/915991455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8538067870 con 0x7f85380724f0 2023-12-04T05:40:44.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.454+0000 7f85357fa640 1 -- 172.21.15.107:0/915991455 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f85200b7040 con 0x7f85380724f0 2023-12-04T05:40:44.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.454+0000 7f85357fa640 1 --2- 172.21.15.107:0/915991455 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f85180668d0 0x7f8518068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:40:44.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.454+0000 7f85357fa640 1 -- 172.21.15.107:0/915991455 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8520122520 con 0x7f85380724f0 2023-12-04T05:40:44.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.454+0000 7f853c982640 1 --2- 172.21.15.107:0/915991455 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f85180668d0 0x7f8518068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:40:44.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.454+0000 7f853c982640 1 --2- 172.21.15.107:0/915991455 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f85180668d0 0x7f8518068d90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8528000f70 tx=0x7f8528009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:40:44.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.458+0000 7f85357fa640 1 -- 172.21.15.107:0/915991455 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f85200ef890 con 0x7f85380724f0 2023-12-04T05:40:44.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.614+0000 7f853ebe6640 1 -- 172.21.15.107:0/915991455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8538151a50 con 0x7f85380724f0 2023-12-04T05:40:44.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.614+0000 7f85357fa640 1 -- 172.21.15.107:0/915991455 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16790 (secure 0 0 0) 0x7f85200f2eb0 con 0x7f85380724f0 2023-12-04T05:40:44.619 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:40:44.619 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":23,"fsid":"603e856b-31ac-475a-8f35-21ab7faadd48","created":"2023-12-04T05:39:34.921690+0000","modified":"2023-12-04T05:40:40.992888+0000","last_up_change":"2023-12-04T05:39:46.235790+0000","last_in_change":"2023-12-04T05:39:37.587639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T05:39:47.567702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T05:39:57.644286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T05:39:58.029897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ab7111c-0ae8-4d41-b6db-e3743614322c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6817","nonce":768714279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6821","nonce":768714279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6829","nonce":768714279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6825","nonce":768714279}]},"public_addr":"172.21.15.107:6817/768714279","cluster_addr":"172.21.15.107:6821/768714279","heartbeat_back_addr":"172.21.15.107:6829/768714279","heartbeat_front_addr":"172.21.15.107:6825/768714279","state":["exists","up"]},{"osd":1,"uuid":"baef7a1f-8026-4fab-9e70-48465271034e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6801","nonce":2496355981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6803","nonce":2496355981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6807","nonce":2496355981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6805","nonce":2496355981}]},"public_addr":"172.21.15.107:6801/2496355981","cluster_addr":"172.21.15.107:6803/2496355981","heartbeat_back_addr":"172.21.15.107:6807/2496355981","heartbeat_front_addr":"172.21.15.107:6805/2496355981","state":["exists","up"]},{"osd":2,"uuid":"90e8cf24-7b54-484a-9d85-b44a9d3dfeab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6809","nonce":3097298224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6811","nonce":3097298224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6815","nonce":3097298224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6813","nonce":3097298224}]},"public_addr":"172.21.15.107:6809/3097298224","cluster_addr":"172.21.15.107:6811/3097298224","heartbeat_back_addr":"172.21.15.107:6815/3097298224","heartbeat_front_addr":"172.21.15.107:6813/3097298224","state":["exists","up"]},{"osd":3,"uuid":"54658731-2b9a-4d5e-ab13-7c1be7d34e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6820","nonce":3773076671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6824","nonce":3773076671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6831","nonce":3773076671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6828","nonce":3773076671}]},"public_addr":"172.21.15.107:6820/3773076671","cluster_addr":"172.21.15.107:6824/3773076671","heartbeat_back_addr":"172.21.15.107:6831/3773076671","heartbeat_front_addr":"172.21.15.107:6828/3773076671","state":["exists","up"]},{"osd":4,"uuid":"9a9dea13-8afe-485d-aa43-45eda691179b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6801","nonce":2747823542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6803","nonce":2747823542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6807","nonce":2747823542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6805","nonce":2747823542}]},"public_addr":"172.21.15.195:6801/2747823542","cluster_addr":"172.21.15.195:6803/2747823542","heartbeat_back_addr":"172.21.15.195:6807/2747823542","heartbeat_front_addr":"172.21.15.195:6805/2747823542","state":["exists","up"]},{"osd":5,"uuid":"db840d83-5a7d-468a-8aa2-1997945847c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6825","nonce":1484264540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6827","nonce":1484264540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6831","nonce":1484264540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6829","nonce":1484264540}]},"public_addr":"172.21.15.195:6825/1484264540","cluster_addr":"172.21.15.195:6827/1484264540","heartbeat_back_addr":"172.21.15.195:6831/1484264540","heartbeat_front_addr":"172.21.15.195:6829/1484264540","state":["exists","up"]},{"osd":6,"uuid":"38b5f165-bf91-467e-a615-4b4784337689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6811","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6813","nonce":2638471412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6817","nonce":2638471412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6823","nonce":2638471412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6821","nonce":2638471412}]},"public_addr":"172.21.15.195:6813/2638471412","cluster_addr":"172.21.15.195:6817/2638471412","heartbeat_back_addr":"172.21.15.195:6823/2638471412","heartbeat_front_addr":"172.21.15.195:6821/2638471412","state":["exists","up"]},{"osd":7,"uuid":"e5b3c596-494b-4df3-9437-bc2d52168ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6809","nonce":768189614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6812","nonce":768189614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6820","nonce":768189614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6816","nonce":768189614}]},"public_addr":"172.21.15.195:6809/768189614","cluster_addr":"172.21.15.195:6812/768189614","heartbeat_back_addr":"172.21.15.195:6820/768189614","heartbeat_front_addr":"172.21.15.195:6816/768189614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.209077+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.086601+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.221568+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.238524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.059477+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.148422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.079867+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.064916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T05:40:44.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.618+0000 7f853ebe6640 1 -- 172.21.15.107:0/915991455 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f85180668d0 msgr2=0x7f8518068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:44.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.618+0000 7f853ebe6640 1 --2- 172.21.15.107:0/915991455 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f85180668d0 0x7f8518068d90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8528000f70 tx=0x7f8528009040 comp rx=0 tx=0).stop 2023-12-04T05:40:44.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.618+0000 7f853ebe6640 1 -- 172.21.15.107:0/915991455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85380724f0 msgr2=0x7f85381528f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:40:44.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.618+0000 7f853ebe6640 1 --2- 172.21.15.107:0/915991455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85380724f0 0x7f85381528f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f85200b3040 tx=0x7f852001b9f0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.618+0000 7f853ebe6640 1 -- 172.21.15.107:0/915991455 shutdown_connections 2023-12-04T05:40:44.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.618+0000 7f853ebe6640 1 --2- 172.21.15.107:0/915991455 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f85180668d0 0x7f8518068d90 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.618+0000 7f853ebe6640 1 --2- 172.21.15.107:0/915991455 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f85380c56f0 0x7f8538152e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.618+0000 7f853ebe6640 1 --2- 172.21.15.107:0/915991455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85380724f0 0x7f85381528f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.618+0000 7f853ebe6640 1 --2- 172.21.15.107:0/915991455 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8538072110 0x7f85381f0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:40:44.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.618+0000 7f853ebe6640 1 -- 172.21.15.107:0/915991455 >> 172.21.15.107:0/915991455 conn(0x7f85380c7ff0 msgr2=0x7f853806d130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:40:44.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.618+0000 7f853ebe6640 1 -- 172.21.15.107:0/915991455 shutdown_connections 2023-12-04T05:40:44.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:40:44.618+0000 7f853ebe6640 1 -- 172.21.15.107:0/915991455 wait complete. 2023-12-04T05:40:44.638 DEBUG:tasks.mds_thrash:Yielding 2023-12-04T05:40:44.638 INFO:teuthology.run_tasks:Running task workunit... 2023-12-04T05:40:44.651 INFO:tasks.workunit:Pulling workunits from ref 47c1c547372ed5bff7984647dc200bac0e019caf 2023-12-04T05:40:44.651 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-04T05:40:44.651 INFO:tasks.workunit:timeout=3h 2023-12-04T05:40:44.651 INFO:tasks.workunit:cleanup=True 2023-12-04T05:40:44.651 DEBUG:teuthology.orchestra.run.smithi107:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-04T05:40:44.652 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2023-12-04T05:40:44.652 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 72 secs before thrashing 2023-12-04T05:40:44.659 INFO:teuthology.orchestra.run.smithi107.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-12-04T05:40:44.659 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2023-12-04T05:40:44.659 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 2dh/45d Inode: 1 Links: 2 2023-12-04T05:40:44.659 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T05:40:44.659 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-12-04T05:40:44.659 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-12-04 05:40:04.753977895 +0000 2023-12-04T05:40:44.659 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-12-04 05:40:38.959310178 +0000 2023-12-04T05:40:44.659 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2023-12-04T05:40:44.659 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-12-04T05:40:44.660 DEBUG:teuthology.orchestra.run.smithi107:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-12-04T05:40:44.723 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 47c1c547372ed5bff7984647dc200bac0e019caf 2023-12-04T05:40:44.770 INFO:tasks.workunit.client.0.smithi107.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-04T05:41:00.034 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:41:00.037 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:41:30.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:41:30.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:41:53.224 INFO:tasks.workunit.client.0.smithi107.stderr:Note: switching to '47c1c547372ed5bff7984647dc200bac0e019caf'. 2023-12-04T05:41:53.224 INFO:tasks.workunit.client.0.smithi107.stderr: 2023-12-04T05:41:53.224 INFO:tasks.workunit.client.0.smithi107.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-04T05:41:53.224 INFO:tasks.workunit.client.0.smithi107.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-04T05:41:53.225 INFO:tasks.workunit.client.0.smithi107.stderr:state without impacting any branches by switching back to a branch. 2023-12-04T05:41:53.225 INFO:tasks.workunit.client.0.smithi107.stderr: 2023-12-04T05:41:53.225 INFO:tasks.workunit.client.0.smithi107.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-04T05:41:53.225 INFO:tasks.workunit.client.0.smithi107.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-04T05:41:53.225 INFO:tasks.workunit.client.0.smithi107.stderr: 2023-12-04T05:41:53.225 INFO:tasks.workunit.client.0.smithi107.stderr: git switch -c 2023-12-04T05:41:53.225 INFO:tasks.workunit.client.0.smithi107.stderr: 2023-12-04T05:41:53.225 INFO:tasks.workunit.client.0.smithi107.stderr:Or undo this operation with: 2023-12-04T05:41:53.225 INFO:tasks.workunit.client.0.smithi107.stderr: 2023-12-04T05:41:53.225 INFO:tasks.workunit.client.0.smithi107.stderr: git switch - 2023-12-04T05:41:53.225 INFO:tasks.workunit.client.0.smithi107.stderr: 2023-12-04T05:41:53.225 INFO:tasks.workunit.client.0.smithi107.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-04T05:41:53.225 INFO:tasks.workunit.client.0.smithi107.stderr: 2023-12-04T05:41:53.225 INFO:tasks.workunit.client.0.smithi107.stderr:HEAD is now at 47c1c547372 Merge PR #51931 into wip-vshankar-testing-20231127.102654 2023-12-04T05:41:53.227 DEBUG:teuthology.orchestra.run.smithi107:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-04T05:41:53.233 INFO:tasks.workunit.client.0.smithi107.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-04T05:41:53.236 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-04T05:41:53.237 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-04T05:41:54.056 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-04T05:41:54.151 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-04T05:41:54.210 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-04T05:41:54.211 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-04T05:41:54.211 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-04T05:41:54.256 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-04T05:41:54.259 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:41:54.259 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-04T05:41:54.304 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2023-12-04T05:41:54.306 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2023-12-04T05:41:54.306 DEBUG:teuthology.orchestra.run.smithi107:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47c1c547372ed5bff7984647dc200bac0e019caf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2023-12-04T05:41:54.364 INFO:tasks.workunit.client.0.smithi107.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2023-12-04T05:41:54.365 INFO:tasks.workunit.client.0.smithi107.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2023-12-04T05:41:54.365 INFO:tasks.workunit.client.0.smithi107.stderr:+ trap -- retry EXIT 2023-12-04T05:41:54.365 INFO:tasks.workunit.client.0.smithi107.stderr:+ rm -rf ffsb 2023-12-04T05:41:54.366 INFO:tasks.workunit.client.0.smithi107.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2023-12-04T05:41:54.373 INFO:tasks.workunit.client.0.smithi107.stderr:Cloning into 'ffsb'... 2023-12-04T05:41:56.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:41:56.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.781+0000 7fe6456a3640 1 Processor -- start 2023-12-04T05:41:56.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.781+0000 7fe6456a3640 1 -- start start 2023-12-04T05:41:56.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.785+0000 7fe6456a3640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe64006e300 0x7fe64006e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:41:56.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.785+0000 7fe6456a3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe6400c8b60 0x7fe6401575a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:41:56.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.785+0000 7fe6456a3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe640157ae0 con 0x7fe640071160 2023-12-04T05:41:56.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.785+0000 7fe6456a3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6400c6d90 con 0x7fe6400c8b60 2023-12-04T05:41:56.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.785+0000 7fe6456a3640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64006ec20 con 0x7fe64006e300 2023-12-04T05:41:56.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.785+0000 7fe63e7fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe6400c8b60 0x7fe6401575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:41:56.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.785+0000 7fe63f7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe640071160 0x7fe64015a160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50002/0 (socket says 172.21.15.107:50002) 2023-12-04T05:41:56.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.785+0000 7fe63f7fe640 1 -- 172.21.15.107:0/250058629 learned_addr learned my addr 172.21.15.107:0/250058629 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:41:56.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.785+0000 7fe63effd640 1 --2- 172.21.15.107:0/250058629 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe64006e300 0x7fe64006e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:41:56.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.785+0000 7fe63e7fc640 1 -- 172.21.15.107:0/250058629 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe64006e300 msgr2=0x7fe64006e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:41:56.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.785+0000 7fe63e7fc640 1 --2- 172.21.15.107:0/250058629 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe64006e300 0x7fe64006e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:56.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.785+0000 7fe63e7fc640 1 -- 172.21.15.107:0/250058629 >> v1:172.21.15.107:6789/0 conn(0x7fe640071160 legacy=0x7fe64015a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:41:56.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.785+0000 7fe63e7fc640 1 -- 172.21.15.107:0/250058629 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64006f340 con 0x7fe6400c8b60 2023-12-04T05:41:56.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.785+0000 7fe63dffb640 1 -- 172.21.15.107:0/250058629 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773563993 0 0) 0x7fe640157ae0 con 0x7fe640071160 2023-12-04T05:41:56.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.785+0000 7fe63e7fc640 1 --2- 172.21.15.107:0/250058629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6400c8b60 0x7fe6401575a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe6340097c0 tx=0x7fe63401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dba89cc046f742f3 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:41:56.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe63dffb640 1 -- 172.21.15.107:0/250058629 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6340a8970 con 0x7fe6400c8b60 2023-12-04T05:41:56.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe63dffb640 1 -- 172.21.15.107:0/250058629 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6340a8ad0 con 0x7fe6400c8b60 2023-12-04T05:41:56.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe63dffb640 1 -- 172.21.15.107:0/250058629 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6340a8e60 con 0x7fe6400c8b60 2023-12-04T05:41:56.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe6456a3640 1 -- 172.21.15.107:0/250058629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6400c8b60 msgr2=0x7fe6401575a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:41:56.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe6456a3640 1 --2- 172.21.15.107:0/250058629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6400c8b60 0x7fe6401575a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe6340097c0 tx=0x7fe63401ec70 comp rx=0 tx=0).stop 2023-12-04T05:41:56.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe6456a3640 1 -- 172.21.15.107:0/250058629 shutdown_connections 2023-12-04T05:41:56.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe6456a3640 1 --2- 172.21.15.107:0/250058629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6400c8b60 0x7fe6401575a0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:56.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe6456a3640 1 --2- 172.21.15.107:0/250058629 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe64006e300 0x7fe64006e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:56.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe6456a3640 1 -- 172.21.15.107:0/250058629 >> 172.21.15.107:0/250058629 conn(0x7fe64006b6c0 msgr2=0x7fe64006bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:41:56.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe6456a3640 1 -- 172.21.15.107:0/250058629 shutdown_connections 2023-12-04T05:41:56.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe6456a3640 1 -- 172.21.15.107:0/250058629 wait complete. 2023-12-04T05:41:56.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe6456a3640 1 Processor -- start 2023-12-04T05:41:56.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe6456a3640 1 -- start start 2023-12-04T05:41:56.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe6456a3640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe64006e300 0x7fe640072820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:41:56.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe6456a3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe640071160 0x7fe6401428a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:41:56.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe6456a3640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6400c8b60 0x7fe64013ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:41:56.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe6456a3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6401422d0 con 0x7fe640071160 2023-12-04T05:41:56.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe6456a3640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6400cb430 con 0x7fe6400c8b60 2023-12-04T05:41:56.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe6456a3640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe640156680 con 0x7fe64006e300 2023-12-04T05:41:56.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe63e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe640071160 0x7fe6401428a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:41:56.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe63e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe640071160 0x7fe6401428a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42654/0 (socket says 172.21.15.107:42654) 2023-12-04T05:41:56.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe63e7fc640 1 -- 172.21.15.107:0/2909236417 learned_addr learned my addr 172.21.15.107:0/2909236417 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:41:56.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe63effd640 1 --2- 172.21.15.107:0/2909236417 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe64006e300 0x7fe640072820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:41:56.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe63f7fe640 1 --2- 172.21.15.107:0/2909236417 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6400c8b60 0x7fe64013ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:41:56.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe63e7fc640 1 -- 172.21.15.107:0/2909236417 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe64006e300 msgr2=0x7fe640072820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:41:56.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe63e7fc640 1 --2- 172.21.15.107:0/2909236417 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe64006e300 0x7fe640072820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:56.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe63e7fc640 1 -- 172.21.15.107:0/2909236417 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6400c8b60 msgr2=0x7fe64013ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:41:56.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe63e7fc640 1 --2- 172.21.15.107:0/2909236417 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6400c8b60 0x7fe64013ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:56.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe63e7fc640 1 -- 172.21.15.107:0/2909236417 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe634009420 con 0x7fe640071160 2023-12-04T05:41:56.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe63f7fe640 1 --2- 172.21.15.107:0/2909236417 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6400c8b60 0x7fe64013ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:41:56.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe63e7fc640 1 --2- 172.21.15.107:0/2909236417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe640071160 0x7fe6401428a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe63401c8a0 tx=0x7fe63401d4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:41:56.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe6277fe640 1 -- 172.21.15.107:0/2909236417 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6340a8970 con 0x7fe640071160 2023-12-04T05:41:56.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe6277fe640 1 -- 172.21.15.107:0/2909236417 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe634006e20 con 0x7fe640071160 2023-12-04T05:41:56.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe6456a3640 1 -- 172.21.15.107:0/2909236417 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6401f0ea0 con 0x7fe640071160 2023-12-04T05:41:56.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe6277fe640 1 -- 172.21.15.107:0/2909236417 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe634004230 con 0x7fe640071160 2023-12-04T05:41:56.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.789+0000 7fe6456a3640 1 -- 172.21.15.107:0/2909236417 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe640156c00 con 0x7fe640071160 2023-12-04T05:41:56.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.793+0000 7fe6456a3640 1 -- 172.21.15.107:0/2909236417 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6400cb5a0 con 0x7fe640071160 2023-12-04T05:41:56.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.793+0000 7fe6277fe640 1 -- 172.21.15.107:0/2909236417 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe634004390 con 0x7fe640071160 2023-12-04T05:41:56.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.793+0000 7fe6277fe640 1 --2- 172.21.15.107:0/2909236417 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe62c066900 0x7fe62c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:41:56.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.793+0000 7fe6277fe640 1 -- 172.21.15.107:0/2909236417 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe634122120 con 0x7fe640071160 2023-12-04T05:41:56.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.793+0000 7fe63effd640 1 --2- 172.21.15.107:0/2909236417 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe62c066900 0x7fe62c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:41:56.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.793+0000 7fe63effd640 1 --2- 172.21.15.107:0/2909236417 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe62c066900 0x7fe62c068dc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe63000b9d0 tx=0x7fe630008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:41:56.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:56.797+0000 7fe6277fe640 1 -- 172.21.15.107:0/2909236417 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe6400cb5a0 con 0x7fe640071160 2023-12-04T05:41:57.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.005+0000 7fe6456a3640 1 -- 172.21.15.107:0/2909236417 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6400cb5a0 con 0x7fe640071160 2023-12-04T05:41:57.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.005+0000 7fe6277fe640 1 -- 172.21.15.107:0/2909236417 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6563 (secure 0 0 0) 0x7fe6400cb5a0 con 0x7fe640071160 2023-12-04T05:41:57.011 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:41:57.011 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:40:28.948661+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":10,"addr":"172.21.15.195:6836/3769196693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":3769196693},{"type":"v1","addr":"172.21.15.195:6836","nonce":3769196693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:41:57.011 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-12-04T05:41:57.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.009+0000 7fe6456a3640 1 -- 172.21.15.107:0/2909236417 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe62c066900 msgr2=0x7fe62c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:41:57.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.009+0000 7fe6456a3640 1 --2- 172.21.15.107:0/2909236417 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe62c066900 0x7fe62c068dc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe63000b9d0 tx=0x7fe630008040 comp rx=0 tx=0).stop 2023-12-04T05:41:57.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.009+0000 7fe6456a3640 1 -- 172.21.15.107:0/2909236417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe640071160 msgr2=0x7fe6401428a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:41:57.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.009+0000 7fe6456a3640 1 --2- 172.21.15.107:0/2909236417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe640071160 0x7fe6401428a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe63401c8a0 tx=0x7fe63401d4b0 comp rx=0 tx=0).stop 2023-12-04T05:41:57.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.009+0000 7fe6456a3640 1 -- 172.21.15.107:0/2909236417 shutdown_connections 2023-12-04T05:41:57.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.009+0000 7fe6456a3640 1 --2- 172.21.15.107:0/2909236417 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe62c066900 0x7fe62c068dc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:57.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.009+0000 7fe6456a3640 1 --2- 172.21.15.107:0/2909236417 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6400c8b60 0x7fe64013ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:57.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.009+0000 7fe6456a3640 1 --2- 172.21.15.107:0/2909236417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe640071160 0x7fe6401428a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:57.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.009+0000 7fe6456a3640 1 --2- 172.21.15.107:0/2909236417 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe64006e300 0x7fe640072820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:57.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.009+0000 7fe6456a3640 1 -- 172.21.15.107:0/2909236417 >> 172.21.15.107:0/2909236417 conn(0x7fe64006b6c0 msgr2=0x7fe640146420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:41:57.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.009+0000 7fe6456a3640 1 -- 172.21.15.107:0/2909236417 shutdown_connections 2023-12-04T05:41:57.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.009+0000 7fe6456a3640 1 -- 172.21.15.107:0/2909236417 wait complete. 2023-12-04T05:41:57.030 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=4) 2023-12-04T05:41:57.030 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-04T05:41:57.030 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T05:41:57.052 INFO:tasks.ceph.mds.b:Stopped 2023-12-04T05:41:57.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:41:57.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.161+0000 7fa03dd0d640 1 Processor -- start 2023-12-04T05:41:57.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.161+0000 7fa03dd0d640 1 -- start start 2023-12-04T05:41:57.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.161+0000 7fa03dd0d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa0380c46a0 0x7fa0380c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:41:57.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.161+0000 7fa03dd0d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa038157180 0x7fa0380c7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:41:57.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.161+0000 7fa03dd0d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa03806e660 con 0x7fa0380c4a80 2023-12-04T05:41:57.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.161+0000 7fa03dd0d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa038156490 con 0x7fa038157180 2023-12-04T05:41:57.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.161+0000 7fa03dd0d640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa038156cf0 con 0x7fa0380c46a0 2023-12-04T05:41:57.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.161+0000 7fa036ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa038157180 0x7fa0380c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:41:57.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.161+0000 7fa037fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa0380c4a80 0x7fa0380c5ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50016/0 (socket says 172.21.15.107:50016) 2023-12-04T05:41:57.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.161+0000 7fa037fff640 1 -- 172.21.15.107:0/2110563523 learned_addr learned my addr 172.21.15.107:0/2110563523 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:41:57.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.161+0000 7fa0377fe640 1 --2- 172.21.15.107:0/2110563523 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa0380c46a0 0x7fa0380c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:41:57.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.161+0000 7fa036ffd640 1 -- 172.21.15.107:0/2110563523 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa0380c46a0 msgr2=0x7fa0380c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:41:57.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.161+0000 7fa036ffd640 1 --2- 172.21.15.107:0/2110563523 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa0380c46a0 0x7fa0380c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:57.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.161+0000 7fa036ffd640 1 -- 172.21.15.107:0/2110563523 >> v1:172.21.15.107:6789/0 conn(0x7fa0380c4a80 legacy=0x7fa0380c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:41:57.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.165+0000 7fa036ffd640 1 -- 172.21.15.107:0/2110563523 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa03806e0a0 con 0x7fa038157180 2023-12-04T05:41:57.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.161+0000 7fa0367fc640 1 -- 172.21.15.107:0/2110563523 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 928832580 0 0) 0x7fa030001120 con 0x7fa0380c4a80 2023-12-04T05:41:57.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.165+0000 7fa036ffd640 1 --2- 172.21.15.107:0/2110563523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa038157180 0x7fa0380c7210 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa02c0097c0 tx=0x7fa02c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65bf75ee62b24fd server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:41:57.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.165+0000 7fa0367fc640 1 -- 172.21.15.107:0/2110563523 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c0ab600 con 0x7fa038157180 2023-12-04T05:41:57.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.165+0000 7fa0367fc640 1 -- 172.21.15.107:0/2110563523 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa02c0a7050 con 0x7fa038157180 2023-12-04T05:41:57.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.165+0000 7fa0367fc640 1 -- 172.21.15.107:0/2110563523 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c0ab980 con 0x7fa038157180 2023-12-04T05:41:57.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.165+0000 7fa03dd0d640 1 -- 172.21.15.107:0/2110563523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa038157180 msgr2=0x7fa0380c7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:41:57.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.165+0000 7fa03dd0d640 1 --2- 172.21.15.107:0/2110563523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa038157180 0x7fa0380c7210 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa02c0097c0 tx=0x7fa02c01c480 comp rx=0 tx=0).stop 2023-12-04T05:41:57.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.165+0000 7fa03dd0d640 1 -- 172.21.15.107:0/2110563523 shutdown_connections 2023-12-04T05:41:57.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.165+0000 7fa03dd0d640 1 --2- 172.21.15.107:0/2110563523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa038157180 0x7fa0380c7210 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:57.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.165+0000 7fa03dd0d640 1 --2- 172.21.15.107:0/2110563523 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa0380c46a0 0x7fa0380c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:57.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.165+0000 7fa03dd0d640 1 -- 172.21.15.107:0/2110563523 >> 172.21.15.107:0/2110563523 conn(0x7fa0380c7ff0 msgr2=0x7fa0380c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:41:57.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.165+0000 7fa03dd0d640 1 -- 172.21.15.107:0/2110563523 shutdown_connections 2023-12-04T05:41:57.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.165+0000 7fa03dd0d640 1 -- 172.21.15.107:0/2110563523 wait complete. 2023-12-04T05:41:57.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa03dd0d640 1 Processor -- start 2023-12-04T05:41:57.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa03dd0d640 1 -- start start 2023-12-04T05:41:57.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa03dd0d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0380c46a0 0x7fa038149f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:41:57.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa03dd0d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa0380c4a80 0x7fa0380cac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:41:57.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa03dd0d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa038157180 0x7fa03813de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:41:57.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa03dd0d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0381f2310 con 0x7fa0380c46a0 2023-12-04T05:41:57.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa03dd0d640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03813e740 con 0x7fa0380c4a80 2023-12-04T05:41:57.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa03dd0d640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa038140980 con 0x7fa038157180 2023-12-04T05:41:57.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa0377fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0380c46a0 0x7fa038149f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:41:57.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa0377fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0380c46a0 0x7fa038149f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42660/0 (socket says 172.21.15.107:42660) 2023-12-04T05:41:57.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa0377fe640 1 -- 172.21.15.107:0/1000879432 learned_addr learned my addr 172.21.15.107:0/1000879432 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:41:57.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa037fff640 1 --2- 172.21.15.107:0/1000879432 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa038157180 0x7fa03813de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:41:57.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa036ffd640 1 --2- 172.21.15.107:0/1000879432 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa0380c4a80 0x7fa0380cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:41:57.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa0377fe640 1 -- 172.21.15.107:0/1000879432 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa038157180 msgr2=0x7fa03813de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:41:57.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa0377fe640 1 --2- 172.21.15.107:0/1000879432 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa038157180 0x7fa03813de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:57.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa0377fe640 1 -- 172.21.15.107:0/1000879432 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa0380c4a80 msgr2=0x7fa0380cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:41:57.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa0377fe640 1 --2- 172.21.15.107:0/1000879432 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa0380c4a80 0x7fa0380cac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:57.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa0377fe640 1 -- 172.21.15.107:0/1000879432 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02c009420 con 0x7fa0380c46a0 2023-12-04T05:41:57.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa036ffd640 1 --2- 172.21.15.107:0/1000879432 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa0380c4a80 0x7fa0380cac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:41:57.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa0377fe640 1 --2- 172.21.15.107:0/1000879432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0380c46a0 0x7fa038149f80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa028007f20 tx=0x7fa02800e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:41:57.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa01ffff640 1 -- 172.21.15.107:0/1000879432 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa028014620 con 0x7fa0380c46a0 2023-12-04T05:41:57.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa01ffff640 1 -- 172.21.15.107:0/1000879432 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa028014780 con 0x7fa0380c46a0 2023-12-04T05:41:57.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa03dd0d640 1 -- 172.21.15.107:0/1000879432 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0381f16d0 con 0x7fa0380c46a0 2023-12-04T05:41:57.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa01ffff640 1 -- 172.21.15.107:0/1000879432 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa028014b30 con 0x7fa0380c46a0 2023-12-04T05:41:57.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa03dd0d640 1 -- 172.21.15.107:0/1000879432 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0381f1c50 con 0x7fa0380c46a0 2023-12-04T05:41:57.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.169+0000 7fa03dd0d640 1 -- 172.21.15.107:0/1000879432 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa038067870 con 0x7fa0380c46a0 2023-12-04T05:41:57.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.173+0000 7fa01ffff640 1 -- 172.21.15.107:0/1000879432 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa028014c90 con 0x7fa0380c46a0 2023-12-04T05:41:57.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.173+0000 7fa01ffff640 1 --2- 172.21.15.107:0/1000879432 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa0240668d0 0x7fa024068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:41:57.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.173+0000 7fa01ffff640 1 -- 172.21.15.107:0/1000879432 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa028085040 con 0x7fa0380c46a0 2023-12-04T05:41:57.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.173+0000 7fa036ffd640 1 --2- 172.21.15.107:0/1000879432 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa0240668d0 0x7fa024068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:41:57.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.173+0000 7fa036ffd640 1 --2- 172.21.15.107:0/1000879432 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa0240668d0 0x7fa024068d90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa02c0ad000 tx=0x7fa02c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:41:57.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.177+0000 7fa01ffff640 1 -- 172.21.15.107:0/1000879432 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa0280b6280 con 0x7fa0380c46a0 2023-12-04T05:41:57.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.373+0000 7fa03dd0d640 1 -- 172.21.15.107:0/1000879432 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa038162c70 con 0x7fa0380c46a0 2023-12-04T05:41:57.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.373+0000 7fa01ffff640 1 -- 172.21.15.107:0/1000879432 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6563 (secure 0 0 0) 0x7fa0280520d0 con 0x7fa0380c46a0 2023-12-04T05:41:57.379 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:41:57.379 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:40:28.948661+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":10,"addr":"172.21.15.195:6836/3769196693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":3769196693},{"type":"v1","addr":"172.21.15.195:6836","nonce":3769196693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:41:57.380 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-12-04T05:41:57.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.377+0000 7fa03dd0d640 1 -- 172.21.15.107:0/1000879432 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa0240668d0 msgr2=0x7fa024068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:41:57.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.377+0000 7fa03dd0d640 1 --2- 172.21.15.107:0/1000879432 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa0240668d0 0x7fa024068d90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa02c0ad000 tx=0x7fa02c008040 comp rx=0 tx=0).stop 2023-12-04T05:41:57.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.377+0000 7fa03dd0d640 1 -- 172.21.15.107:0/1000879432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0380c46a0 msgr2=0x7fa038149f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:41:57.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.377+0000 7fa03dd0d640 1 --2- 172.21.15.107:0/1000879432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0380c46a0 0x7fa038149f80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa028007f20 tx=0x7fa02800e570 comp rx=0 tx=0).stop 2023-12-04T05:41:57.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.377+0000 7fa03dd0d640 1 -- 172.21.15.107:0/1000879432 shutdown_connections 2023-12-04T05:41:57.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.377+0000 7fa03dd0d640 1 --2- 172.21.15.107:0/1000879432 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa0240668d0 0x7fa024068d90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:57.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.377+0000 7fa03dd0d640 1 --2- 172.21.15.107:0/1000879432 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa038157180 0x7fa03813de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:57.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.377+0000 7fa03dd0d640 1 --2- 172.21.15.107:0/1000879432 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa0380c4a80 0x7fa0380cac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:57.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.377+0000 7fa03dd0d640 1 --2- 172.21.15.107:0/1000879432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0380c46a0 0x7fa038149f80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:57.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.377+0000 7fa03dd0d640 1 -- 172.21.15.107:0/1000879432 >> 172.21.15.107:0/1000879432 conn(0x7fa0380c7ff0 msgr2=0x7fa0380c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:41:57.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.377+0000 7fa03dd0d640 1 -- 172.21.15.107:0/1000879432 shutdown_connections 2023-12-04T05:41:57.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:57.377+0000 7fa03dd0d640 1 -- 172.21.15.107:0/1000879432 wait complete. 2023-12-04T05:41:57.398 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-12-04T05:41:59.400 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:41:59.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.525+0000 7fec06a50640 1 Processor -- start 2023-12-04T05:41:59.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.525+0000 7fec06a50640 1 -- start start 2023-12-04T05:41:59.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7fec06a50640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fec000c46a0 0x7fec000c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:41:59.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7fec06a50640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fec000c4a80 0x7fec000c7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:41:59.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7fec06a50640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec0006e660 con 0x7fec00157180 2023-12-04T05:41:59.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7fec06a50640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec00156490 con 0x7fec000c46a0 2023-12-04T05:41:59.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7fec06a50640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec00156cf0 con 0x7fec000c4a80 2023-12-04T05:41:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7febff7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fec00157180 0x7fec000c5ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56836/0 (socket says 172.21.15.107:56836) 2023-12-04T05:41:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7febff7fe640 1 -- 172.21.15.107:0/321844009 learned_addr learned my addr 172.21.15.107:0/321844009 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:41:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7febfffff640 1 --2- 172.21.15.107:0/321844009 >> v2:172.21.15.195:3300/0 conn(0x7fec000c46a0 0x7fec000c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:41:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7fec04fed640 1 --2- 172.21.15.107:0/321844009 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fec000c4a80 0x7fec000c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:41:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7febfeffd640 1 -- 172.21.15.107:0/321844009 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166362638 0 0) 0x7febe8001120 con 0x7fec00157180 2023-12-04T05:41:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7febfeffd640 1 -- 172.21.15.107:0/321844009 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febec003530 con 0x7fec00157180 2023-12-04T05:41:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7febfeffd640 1 -- 172.21.15.107:0/321844009 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3314590928 0 0) 0x7febec003530 con 0x7fec00157180 2023-12-04T05:41:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7febfeffd640 1 -- 172.21.15.107:0/321844009 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fec000c4a80 msgr2=0x7fec000c7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:41:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7febfeffd640 1 --2- 172.21.15.107:0/321844009 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fec000c4a80 0x7fec000c7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7febfeffd640 1 -- 172.21.15.107:0/321844009 >> v2:172.21.15.195:3300/0 conn(0x7fec000c46a0 msgr2=0x7fec000c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:41:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7febfeffd640 1 --2- 172.21.15.107:0/321844009 >> v2:172.21.15.195:3300/0 conn(0x7fec000c46a0 0x7fec000c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7febfeffd640 1 -- 172.21.15.107:0/321844009 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0006e0a0 con 0x7fec00157180 2023-12-04T05:41:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7febfeffd640 1 -- 172.21.15.107:0/321844009 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7febe8001810 con 0x7fec00157180 2023-12-04T05:41:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7febfeffd640 1 -- 172.21.15.107:0/321844009 >> v1:172.21.15.107:6789/0 conn(0x7fec00157180 legacy=0x7fec000c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:41:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7febfeffd640 1 --2- 172.21.15.107:0/321844009 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7febec003980 0x7febec003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:41:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7febfeffd640 1 --2- 172.21.15.107:0/321844009 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fec000c4a80 0x7febec0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:41:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7febfeffd640 1 --2- 172.21.15.107:0/321844009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febec004d10 0x7febec005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:41:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7febfeffd640 1 -- 172.21.15.107:0/321844009 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec00156490 con 0x7febec004d10 2023-12-04T05:41:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7febfeffd640 1 -- 172.21.15.107:0/321844009 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec00156cf0 con 0x7febec003980 2023-12-04T05:41:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7febfeffd640 1 -- 172.21.15.107:0/321844009 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febec00b990 con 0x7fec000c4a80 2023-12-04T05:41:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7fec04fed640 1 --2- 172.21.15.107:0/321844009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febec004d10 0x7febec005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:41:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7fec04fed640 1 -- 172.21.15.107:0/321844009 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fec000c4a80 msgr2=0x7febec0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:41:59.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7fec04fed640 1 --2- 172.21.15.107:0/321844009 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fec000c4a80 0x7febec0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:59.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7fec04fed640 1 -- 172.21.15.107:0/321844009 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7febec003980 msgr2=0x7febec003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:41:59.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7fec04fed640 1 --2- 172.21.15.107:0/321844009 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7febec003980 0x7febec003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:59.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.529+0000 7fec04fed640 1 -- 172.21.15.107:0/321844009 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febec003530 con 0x7febec004d10 2023-12-04T05:41:59.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec04fed640 1 --2- 172.21.15.107:0/321844009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febec004d10 0x7febec005190 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7febf8009fc0 tx=0x7febf801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29e14b8964c7e081 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:41:59.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7febfeffd640 1 -- 172.21.15.107:0/321844009 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf80a5cb0 con 0x7febec004d10 2023-12-04T05:41:59.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7febfeffd640 1 -- 172.21.15.107:0/321844009 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf800f040 con 0x7febec004d10 2023-12-04T05:41:59.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7febfeffd640 1 -- 172.21.15.107:0/321844009 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf80ad680 con 0x7febec004d10 2023-12-04T05:41:59.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec06a50640 1 -- 172.21.15.107:0/321844009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febec004d10 msgr2=0x7febec005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:41:59.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec06a50640 1 --2- 172.21.15.107:0/321844009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febec004d10 0x7febec005190 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7febf8009fc0 tx=0x7febf801cca0 comp rx=0 tx=0).stop 2023-12-04T05:41:59.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec04fed640 1 -- 172.21.15.107:0/321844009 reap_dead start 2023-12-04T05:41:59.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec06a50640 1 -- 172.21.15.107:0/321844009 shutdown_connections 2023-12-04T05:41:59.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec06a50640 1 -- 172.21.15.107:0/321844009 >> 172.21.15.107:0/321844009 conn(0x7fec000c7ff0 msgr2=0x7fec000c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:41:59.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec06a50640 1 -- 172.21.15.107:0/321844009 shutdown_connections 2023-12-04T05:41:59.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec06a50640 1 -- 172.21.15.107:0/321844009 wait complete. 2023-12-04T05:41:59.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec06a50640 1 Processor -- start 2023-12-04T05:41:59.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec06a50640 1 -- start start 2023-12-04T05:41:59.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec06a50640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7febec004d10 0x7fec001eb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:41:59.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec06a50640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec0014e560 0x7fec0014d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:41:59.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec06a50640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec00148b80 0x7fec00145ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:41:59.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec06a50640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec000c6800 con 0x7fec00148b80 2023-12-04T05:41:59.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec06a50640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec001ebf10 con 0x7fec0014e560 2023-12-04T05:41:59.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec06a50640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec0014cb80 con 0x7febec004d10 2023-12-04T05:41:59.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec04fed640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec00148b80 0x7fec00145ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:41:59.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec04fed640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec00148b80 0x7fec00145ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58668/0 (socket says 172.21.15.107:58668) 2023-12-04T05:41:59.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec04fed640 1 -- 172.21.15.107:0/2911953268 learned_addr learned my addr 172.21.15.107:0/2911953268 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:41:59.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7febff7fe640 1 --2- 172.21.15.107:0/2911953268 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec0014e560 0x7fec0014d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:41:59.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7febfffff640 1 --2- 172.21.15.107:0/2911953268 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7febec004d10 0x7fec001eb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:41:59.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec04fed640 1 -- 172.21.15.107:0/2911953268 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7febec004d10 msgr2=0x7fec001eb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:41:59.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec04fed640 1 --2- 172.21.15.107:0/2911953268 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7febec004d10 0x7fec001eb9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:59.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec04fed640 1 -- 172.21.15.107:0/2911953268 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec0014e560 msgr2=0x7fec0014d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:41:59.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec04fed640 1 --2- 172.21.15.107:0/2911953268 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec0014e560 0x7fec0014d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:59.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec04fed640 1 -- 172.21.15.107:0/2911953268 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec000c6ff0 con 0x7fec00148b80 2023-12-04T05:41:59.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7febff7fe640 1 --2- 172.21.15.107:0/2911953268 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec0014e560 0x7fec0014d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:41:59.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec04fed640 1 --2- 172.21.15.107:0/2911953268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec00148b80 0x7fec00145ab0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7febf8002240 tx=0x7febf8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:41:59.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7febfcff9640 1 -- 172.21.15.107:0/2911953268 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf80039a0 con 0x7fec00148b80 2023-12-04T05:41:59.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7febfcff9640 1 -- 172.21.15.107:0/2911953268 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf8003b00 con 0x7fec00148b80 2023-12-04T05:41:59.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec06a50640 1 -- 172.21.15.107:0/2911953268 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec000c6a30 con 0x7fec00148b80 2023-12-04T05:41:59.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7febfcff9640 1 -- 172.21.15.107:0/2911953268 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf8003e70 con 0x7fec00148b80 2023-12-04T05:41:59.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.533+0000 7fec06a50640 1 -- 172.21.15.107:0/2911953268 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec0014c530 con 0x7fec00148b80 2023-12-04T05:41:59.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.537+0000 7fec06a50640 1 -- 172.21.15.107:0/2911953268 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec00067870 con 0x7fec00148b80 2023-12-04T05:41:59.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.537+0000 7febfcff9640 1 -- 172.21.15.107:0/2911953268 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7febf80b4040 con 0x7fec00148b80 2023-12-04T05:41:59.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.537+0000 7febfcff9640 1 --2- 172.21.15.107:0/2911953268 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7febe0066900 0x7febe0068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:41:59.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.537+0000 7febfcff9640 1 -- 172.21.15.107:0/2911953268 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7febf811e3b0 con 0x7fec00148b80 2023-12-04T05:41:59.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.537+0000 7febfffff640 1 --2- 172.21.15.107:0/2911953268 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7febe0066900 0x7febe0068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:41:59.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.537+0000 7febfffff640 1 --2- 172.21.15.107:0/2911953268 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7febe0066900 0x7febe0068dc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7febf00047c0 tx=0x7febf0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:41:59.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.541+0000 7febfcff9640 1 -- 172.21.15.107:0/2911953268 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7febf80eb590 con 0x7fec00148b80 2023-12-04T05:41:59.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.733+0000 7fec06a50640 1 -- 172.21.15.107:0/2911953268 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec0014ccc0 con 0x7fec00148b80 2023-12-04T05:41:59.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.733+0000 7febfcff9640 1 -- 172.21.15.107:0/2911953268 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6563 (secure 0 0 0) 0x7febf80bd090 con 0x7fec00148b80 2023-12-04T05:41:59.740 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:41:59.740 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:40:28.948661+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":10,"addr":"172.21.15.195:6836/3769196693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":3769196693},{"type":"v1","addr":"172.21.15.195:6836","nonce":3769196693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:41:59.740 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-12-04T05:41:59.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.737+0000 7fec06a50640 1 -- 172.21.15.107:0/2911953268 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7febe0066900 msgr2=0x7febe0068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:41:59.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.737+0000 7fec06a50640 1 --2- 172.21.15.107:0/2911953268 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7febe0066900 0x7febe0068dc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7febf00047c0 tx=0x7febf0008040 comp rx=0 tx=0).stop 2023-12-04T05:41:59.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.737+0000 7fec06a50640 1 -- 172.21.15.107:0/2911953268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec00148b80 msgr2=0x7fec00145ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:41:59.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.737+0000 7fec06a50640 1 --2- 172.21.15.107:0/2911953268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec00148b80 0x7fec00145ab0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7febf8002240 tx=0x7febf8009750 comp rx=0 tx=0).stop 2023-12-04T05:41:59.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.737+0000 7fec06a50640 1 -- 172.21.15.107:0/2911953268 shutdown_connections 2023-12-04T05:41:59.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.737+0000 7fec06a50640 1 --2- 172.21.15.107:0/2911953268 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7febe0066900 0x7febe0068dc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:59.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.737+0000 7fec06a50640 1 --2- 172.21.15.107:0/2911953268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec00148b80 0x7fec00145ab0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:59.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.737+0000 7fec06a50640 1 --2- 172.21.15.107:0/2911953268 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fec0014e560 0x7fec0014d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:59.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.737+0000 7fec06a50640 1 --2- 172.21.15.107:0/2911953268 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7febec004d10 0x7fec001eb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:41:59.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.737+0000 7fec06a50640 1 -- 172.21.15.107:0/2911953268 >> 172.21.15.107:0/2911953268 conn(0x7fec000c7ff0 msgr2=0x7fec0014b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:41:59.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.737+0000 7fec06a50640 1 -- 172.21.15.107:0/2911953268 shutdown_connections 2023-12-04T05:41:59.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:41:59.737+0000 7fec06a50640 1 -- 172.21.15.107:0/2911953268 wait complete. 2023-12-04T05:41:59.759 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-12-04T05:42:00.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:42:00.199 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:42:01.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:42:01.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.889+0000 7f4432f53640 1 Processor -- start 2023-12-04T05:42:01.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.889+0000 7f4432f53640 1 -- start start 2023-12-04T05:42:01.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.893+0000 7f4432f53640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f442c150dc0 0x7f442c15a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:01.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.893+0000 7f4432f53640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f442c071080 0x7f442c15ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:01.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.893+0000 7f4432f53640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f442c15a390 con 0x7f442c156b70 2023-12-04T05:42:01.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.893+0000 7f4432f53640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f442c157ba0 con 0x7f442c071080 2023-12-04T05:42:01.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.893+0000 7f4432f53640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f442c157040 con 0x7f442c150dc0 2023-12-04T05:42:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.893+0000 7f44314f0640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f442c071080 0x7f442c15ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.893+0000 7f4430cef640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f442c150dc0 0x7f442c15a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.893+0000 7f442bfff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f442c156b70 0x7f442c071740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:36518/0 (socket says 172.21.15.107:36518) 2023-12-04T05:42:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.893+0000 7f44314f0640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f442c071080 0x7f442c15ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.107:37944/0 (socket says 172.21.15.107:37944) 2023-12-04T05:42:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.893+0000 7f442bfff640 1 -- 172.21.15.107:0/232182583 learned_addr learned my addr 172.21.15.107:0/232182583 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:42:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.893+0000 7f4430cef640 1 -- 172.21.15.107:0/232182583 >> v2:172.21.15.195:3300/0 conn(0x7f442c071080 msgr2=0x7f442c15ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.893+0000 7f4430cef640 1 --2- 172.21.15.107:0/232182583 >> v2:172.21.15.195:3300/0 conn(0x7f442c071080 0x7f442c15ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.893+0000 7f4430cef640 1 -- 172.21.15.107:0/232182583 >> v1:172.21.15.195:6789/0 conn(0x7f442c156b70 legacy=0x7f442c071740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.893+0000 7f4430cef640 1 -- 172.21.15.107:0/232182583 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442c157300 con 0x7f442c150dc0 2023-12-04T05:42:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f4430cef640 1 --2- 172.21.15.107:0/232182583 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f442c150dc0 0x7f442c15a820 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f44200097c0 tx=0x7f442001c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf87739acd53212b server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f442b7fe640 1 -- 172.21.15.107:0/232182583 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44200ab680 con 0x7f442c150dc0 2023-12-04T05:42:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f442b7fe640 1 -- 172.21.15.107:0/232182583 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44200ab7e0 con 0x7f442c150dc0 2023-12-04T05:42:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f442b7fe640 1 -- 172.21.15.107:0/232182583 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44200abb70 con 0x7f442c150dc0 2023-12-04T05:42:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f4432f53640 1 -- 172.21.15.107:0/232182583 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f442c150dc0 msgr2=0x7f442c15a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f4432f53640 1 --2- 172.21.15.107:0/232182583 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f442c150dc0 0x7f442c15a820 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f44200097c0 tx=0x7f442001c480 comp rx=0 tx=0).stop 2023-12-04T05:42:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f4432f53640 1 -- 172.21.15.107:0/232182583 shutdown_connections 2023-12-04T05:42:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f4432f53640 1 --2- 172.21.15.107:0/232182583 >> v2:172.21.15.195:3300/0 conn(0x7f442c071080 0x7f442c15ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f4432f53640 1 --2- 172.21.15.107:0/232182583 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f442c150dc0 0x7f442c15a820 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f4432f53640 1 -- 172.21.15.107:0/232182583 >> 172.21.15.107:0/232182583 conn(0x7f442c0cc490 msgr2=0x7f442c0cc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f4432f53640 1 -- 172.21.15.107:0/232182583 shutdown_connections 2023-12-04T05:42:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f4432f53640 1 -- 172.21.15.107:0/232182583 wait complete. 2023-12-04T05:42:01.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f4432f53640 1 Processor -- start 2023-12-04T05:42:01.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f4432f53640 1 -- start start 2023-12-04T05:42:01.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f4432f53640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f442c071080 0x7f442c15ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:01.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f4432f53640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f442c150dc0 0x7f442c1549f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:01.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f4432f53640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f442c156b70 0x7f442c1f37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:01.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f4432f53640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442c155ba0 con 0x7f442c071080 2023-12-04T05:42:01.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f4432f53640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442c14cb80 con 0x7f442c156b70 2023-12-04T05:42:01.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f4432f53640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f442c1f6d60 con 0x7f442c150dc0 2023-12-04T05:42:01.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f4430cef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f442c071080 0x7f442c15ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:01.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f4430cef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f442c071080 0x7f442c15ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58670/0 (socket says 172.21.15.107:58670) 2023-12-04T05:42:01.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f4430cef640 1 -- 172.21.15.107:0/4084853453 learned_addr learned my addr 172.21.15.107:0/4084853453 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:01.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f442bfff640 1 --2- 172.21.15.107:0/4084853453 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f442c150dc0 0x7f442c1549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:01.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f44314f0640 1 --2- 172.21.15.107:0/4084853453 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f442c156b70 0x7f442c1f37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:01.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f4430cef640 1 -- 172.21.15.107:0/4084853453 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f442c150dc0 msgr2=0x7f442c1549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:01.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.897+0000 7f4430cef640 1 --2- 172.21.15.107:0/4084853453 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f442c150dc0 0x7f442c1549f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:01.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.901+0000 7f4430cef640 1 -- 172.21.15.107:0/4084853453 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f442c156b70 msgr2=0x7f442c1f37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:01.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.901+0000 7f4430cef640 1 --2- 172.21.15.107:0/4084853453 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f442c156b70 0x7f442c1f37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:01.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.901+0000 7f4430cef640 1 -- 172.21.15.107:0/4084853453 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4420009420 con 0x7f442c071080 2023-12-04T05:42:01.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.901+0000 7f442bfff640 1 --2- 172.21.15.107:0/4084853453 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f442c150dc0 0x7f442c1549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:42:01.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.901+0000 7f4430cef640 1 --2- 172.21.15.107:0/4084853453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f442c071080 0x7f442c15ffd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f442001a0b0 tx=0x7f442001acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:01.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.901+0000 7f44297fa640 1 -- 172.21.15.107:0/4084853453 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44200bf610 con 0x7f442c071080 2023-12-04T05:42:01.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.901+0000 7f4432f53640 1 -- 172.21.15.107:0/4084853453 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f442c0c44e0 con 0x7f442c071080 2023-12-04T05:42:01.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.901+0000 7f44297fa640 1 -- 172.21.15.107:0/4084853453 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44200bf770 con 0x7f442c071080 2023-12-04T05:42:01.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.901+0000 7f44297fa640 1 -- 172.21.15.107:0/4084853453 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44200bfae0 con 0x7f442c071080 2023-12-04T05:42:01.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.901+0000 7f4432f53640 1 -- 172.21.15.107:0/4084853453 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f442c1f7600 con 0x7f442c071080 2023-12-04T05:42:01.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.901+0000 7f4432f53640 1 -- 172.21.15.107:0/4084853453 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f442c067ec0 con 0x7f442c071080 2023-12-04T05:42:01.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.901+0000 7f44297fa640 1 -- 172.21.15.107:0/4084853453 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f44200bfc40 con 0x7f442c071080 2023-12-04T05:42:01.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.901+0000 7f44297fa640 1 --2- 172.21.15.107:0/4084853453 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f44180668d0 0x7f4418068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:01.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.901+0000 7f44297fa640 1 -- 172.21.15.107:0/4084853453 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f442011f050 con 0x7f442c071080 2023-12-04T05:42:01.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.901+0000 7f442bfff640 1 --2- 172.21.15.107:0/4084853453 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f44180668d0 0x7f4418068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:01.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.901+0000 7f442bfff640 1 --2- 172.21.15.107:0/4084853453 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f44180668d0 0x7f4418068d90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f442c0c3160 tx=0x7f441c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:01.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:01.905+0000 7f44297fa640 1 -- 172.21.15.107:0/4084853453 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f44200ec100 con 0x7f442c071080 2023-12-04T05:42:02.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:02.101+0000 7f4432f53640 1 -- 172.21.15.107:0/4084853453 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f442c0c7b50 con 0x7f442c071080 2023-12-04T05:42:02.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:02.105+0000 7f44297fa640 1 -- 172.21.15.107:0/4084853453 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6563 (secure 0 0 0) 0x7f44200ef720 con 0x7f442c071080 2023-12-04T05:42:02.109 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:42:02.109 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:40:28.948661+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":10,"addr":"172.21.15.195:6836/3769196693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":3769196693},{"type":"v1","addr":"172.21.15.195:6836","nonce":3769196693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:42:02.109 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-12-04T05:42:02.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:02.105+0000 7f4432f53640 1 -- 172.21.15.107:0/4084853453 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f44180668d0 msgr2=0x7f4418068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:02.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:02.105+0000 7f4432f53640 1 --2- 172.21.15.107:0/4084853453 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f44180668d0 0x7f4418068d90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f442c0c3160 tx=0x7f441c009040 comp rx=0 tx=0).stop 2023-12-04T05:42:02.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:02.105+0000 7f4432f53640 1 -- 172.21.15.107:0/4084853453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f442c071080 msgr2=0x7f442c15ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:02.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:02.105+0000 7f4432f53640 1 --2- 172.21.15.107:0/4084853453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f442c071080 0x7f442c15ffd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f442001a0b0 tx=0x7f442001acc0 comp rx=0 tx=0).stop 2023-12-04T05:42:02.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:02.105+0000 7f4432f53640 1 -- 172.21.15.107:0/4084853453 shutdown_connections 2023-12-04T05:42:02.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:02.105+0000 7f4432f53640 1 --2- 172.21.15.107:0/4084853453 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f44180668d0 0x7f4418068d90 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:02.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:02.105+0000 7f4432f53640 1 --2- 172.21.15.107:0/4084853453 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f442c156b70 0x7f442c1f37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:02.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:02.105+0000 7f4432f53640 1 --2- 172.21.15.107:0/4084853453 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f442c150dc0 0x7f442c1549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:02.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:02.105+0000 7f4432f53640 1 --2- 172.21.15.107:0/4084853453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f442c071080 0x7f442c15ffd0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:02.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:02.105+0000 7f4432f53640 1 -- 172.21.15.107:0/4084853453 >> 172.21.15.107:0/4084853453 conn(0x7f442c0cc490 msgr2=0x7f442c0c4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:02.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:02.105+0000 7f4432f53640 1 -- 172.21.15.107:0/4084853453 shutdown_connections 2023-12-04T05:42:02.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:02.105+0000 7f4432f53640 1 -- 172.21.15.107:0/4084853453 wait complete. 2023-12-04T05:42:02.128 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-12-04T05:42:04.130 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:42:04.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.269+0000 7f5377015640 1 Processor -- start 2023-12-04T05:42:04.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.269+0000 7f5377015640 1 -- start start 2023-12-04T05:42:04.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.273+0000 7f5377015640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f53700ccd90 0x7f537006be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:04.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.273+0000 7f5377015640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f537006c340 0x7f53700cb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:04.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.273+0000 7f5377015640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f537015e310 con 0x7f5370153e90 2023-12-04T05:42:04.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.273+0000 7f5377015640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53700c6100 con 0x7f537006c340 2023-12-04T05:42:04.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.273+0000 7f5377015640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53700cd1d0 con 0x7f53700ccd90 2023-12-04T05:42:04.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.273+0000 7f53755b2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f537006c340 0x7f53700cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:04.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.273+0000 7f53755b2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f537006c340 0x7f53700cb030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58684/0 (socket says 172.21.15.107:58684) 2023-12-04T05:42:04.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.273+0000 7f53755b2640 1 -- 172.21.15.107:0/426001056 learned_addr learned my addr 172.21.15.107:0/426001056 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:04.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.273+0000 7f536ffff640 1 --2- 172.21.15.107:0/426001056 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f53700ccd90 0x7f537006be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:04.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.273+0000 7f53755b2640 1 -- 172.21.15.107:0/426001056 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f53700ccd90 msgr2=0x7f537006be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:04.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.273+0000 7f53755b2640 1 --2- 172.21.15.107:0/426001056 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f53700ccd90 0x7f537006be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:04.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.273+0000 7f53755b2640 1 -- 172.21.15.107:0/426001056 >> v1:172.21.15.195:6789/0 conn(0x7f5370153e90 legacy=0x7f537015a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:04.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.273+0000 7f53755b2640 1 -- 172.21.15.107:0/426001056 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5370154270 con 0x7f537006c340 2023-12-04T05:42:04.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.273+0000 7f536ffff640 1 --2- 172.21.15.107:0/426001056 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f53700ccd90 0x7f537006be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:42:04.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.273+0000 7f53755b2640 1 --2- 172.21.15.107:0/426001056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f537006c340 0x7f53700cb030 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5368009e50 tx=0x7f536801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f34bcc049d53127f server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:04.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.273+0000 7f536f7fe640 1 -- 172.21.15.107:0/426001056 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53680af400 con 0x7f537006c340 2023-12-04T05:42:04.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.273+0000 7f536f7fe640 1 -- 172.21.15.107:0/426001056 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53680ab050 con 0x7f537006c340 2023-12-04T05:42:04.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.273+0000 7f536f7fe640 1 -- 172.21.15.107:0/426001056 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53680af780 con 0x7f537006c340 2023-12-04T05:42:04.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f5377015640 1 -- 172.21.15.107:0/426001056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f537006c340 msgr2=0x7f53700cb030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:04.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f5377015640 1 --2- 172.21.15.107:0/426001056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f537006c340 0x7f53700cb030 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5368009e50 tx=0x7f536801f1f0 comp rx=0 tx=0).stop 2023-12-04T05:42:04.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f5377015640 1 -- 172.21.15.107:0/426001056 shutdown_connections 2023-12-04T05:42:04.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f5377015640 1 --2- 172.21.15.107:0/426001056 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f53700ccd90 0x7f537006be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:04.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f5377015640 1 --2- 172.21.15.107:0/426001056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f537006c340 0x7f53700cb030 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:04.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f5377015640 1 -- 172.21.15.107:0/426001056 >> 172.21.15.107:0/426001056 conn(0x7f537006b790 msgr2=0x7f53700663d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:04.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f5377015640 1 -- 172.21.15.107:0/426001056 shutdown_connections 2023-12-04T05:42:04.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f5377015640 1 -- 172.21.15.107:0/426001056 wait complete. 2023-12-04T05:42:04.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f5377015640 1 Processor -- start 2023-12-04T05:42:04.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f5377015640 1 -- start start 2023-12-04T05:42:04.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f5377015640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f537006c340 0x7f5370146520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:04.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f5377015640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f53700ccd90 0x7f53700b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:04.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f5377015640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5370153e90 0x7f53700b3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:04.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f5377015640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537015c4f0 con 0x7f5370153e90 2023-12-04T05:42:04.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f5377015640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537014ea50 con 0x7f53700ccd90 2023-12-04T05:42:04.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f5377015640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f537015c790 con 0x7f537006c340 2023-12-04T05:42:04.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f53755b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5370153e90 0x7f53700b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:04.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f53755b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5370153e90 0x7f53700b3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58692/0 (socket says 172.21.15.107:58692) 2023-12-04T05:42:04.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f53755b2640 1 -- 172.21.15.107:0/3571466639 learned_addr learned my addr 172.21.15.107:0/3571466639 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:04.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f5374db1640 1 --2- 172.21.15.107:0/3571466639 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f537006c340 0x7f5370146520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:04.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f536ffff640 1 --2- 172.21.15.107:0/3571466639 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f53700ccd90 0x7f53700b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:04.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f53755b2640 1 -- 172.21.15.107:0/3571466639 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f537006c340 msgr2=0x7f5370146520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:04.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f53755b2640 1 --2- 172.21.15.107:0/3571466639 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f537006c340 0x7f5370146520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:04.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f53755b2640 1 -- 172.21.15.107:0/3571466639 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f53700ccd90 msgr2=0x7f53700b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:04.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f53755b2640 1 --2- 172.21.15.107:0/3571466639 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f53700ccd90 0x7f53700b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:04.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f53755b2640 1 -- 172.21.15.107:0/3571466639 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5368009b30 con 0x7f5370153e90 2023-12-04T05:42:04.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f53755b2640 1 --2- 172.21.15.107:0/3571466639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5370153e90 0x7f53700b3250 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f536801ce20 tx=0x7f536801bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:04.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f536d7fa640 1 -- 172.21.15.107:0/3571466639 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53680c0070 con 0x7f5370153e90 2023-12-04T05:42:04.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f536d7fa640 1 -- 172.21.15.107:0/3571466639 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5368009510 con 0x7f5370153e90 2023-12-04T05:42:04.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f5377015640 1 -- 172.21.15.107:0/3571466639 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f537014fd60 con 0x7f5370153e90 2023-12-04T05:42:04.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f536d7fa640 1 -- 172.21.15.107:0/3571466639 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53680afe40 con 0x7f5370153e90 2023-12-04T05:42:04.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.277+0000 7f5377015640 1 -- 172.21.15.107:0/3571466639 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5370150320 con 0x7f5370153e90 2023-12-04T05:42:04.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.281+0000 7f5377015640 1 -- 172.21.15.107:0/3571466639 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5370067ec0 con 0x7f5370153e90 2023-12-04T05:42:04.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.281+0000 7f536d7fa640 1 -- 172.21.15.107:0/3571466639 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f53680b6040 con 0x7f5370153e90 2023-12-04T05:42:04.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.281+0000 7f536d7fa640 1 --2- 172.21.15.107:0/3571466639 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f53540668d0 0x7f5354068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:04.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.281+0000 7f536d7fa640 1 -- 172.21.15.107:0/3571466639 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5368122260 con 0x7f5370153e90 2023-12-04T05:42:04.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.281+0000 7f5374db1640 1 --2- 172.21.15.107:0/3571466639 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f53540668d0 0x7f5354068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:04.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.281+0000 7f5374db1640 1 --2- 172.21.15.107:0/3571466639 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f53540668d0 0x7f5354068d90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5364000f70 tx=0x7f5364009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:04.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.285+0000 7f536d7fa640 1 -- 172.21.15.107:0/3571466639 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f53680ef3b0 con 0x7f5370153e90 2023-12-04T05:42:04.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.477+0000 7f5377015640 1 -- 172.21.15.107:0/3571466639 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53700c9bf0 con 0x7f5370153e90 2023-12-04T05:42:04.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.477+0000 7f536d7fa640 1 -- 172.21.15.107:0/3571466639 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6563 (secure 0 0 0) 0x7f53680f29d0 con 0x7f5370153e90 2023-12-04T05:42:04.483 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:42:04.483 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:40:28.948661+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":10,"addr":"172.21.15.195:6836/3769196693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":3769196693},{"type":"v1","addr":"172.21.15.195:6836","nonce":3769196693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:42:04.483 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-12-04T05:42:04.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.481+0000 7f5377015640 1 -- 172.21.15.107:0/3571466639 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f53540668d0 msgr2=0x7f5354068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:04.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.481+0000 7f5377015640 1 --2- 172.21.15.107:0/3571466639 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f53540668d0 0x7f5354068d90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5364000f70 tx=0x7f5364009040 comp rx=0 tx=0).stop 2023-12-04T05:42:04.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.481+0000 7f5377015640 1 -- 172.21.15.107:0/3571466639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5370153e90 msgr2=0x7f53700b3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:04.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.481+0000 7f5377015640 1 --2- 172.21.15.107:0/3571466639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5370153e90 0x7f53700b3250 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f536801ce20 tx=0x7f536801bf90 comp rx=0 tx=0).stop 2023-12-04T05:42:04.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.481+0000 7f5377015640 1 -- 172.21.15.107:0/3571466639 shutdown_connections 2023-12-04T05:42:04.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.481+0000 7f5377015640 1 --2- 172.21.15.107:0/3571466639 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f53540668d0 0x7f5354068d90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:04.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.481+0000 7f5377015640 1 --2- 172.21.15.107:0/3571466639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5370153e90 0x7f53700b3250 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:04.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.481+0000 7f5377015640 1 --2- 172.21.15.107:0/3571466639 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f53700ccd90 0x7f53700b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:04.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.481+0000 7f5377015640 1 --2- 172.21.15.107:0/3571466639 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f537006c340 0x7f5370146520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:04.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.481+0000 7f5377015640 1 -- 172.21.15.107:0/3571466639 >> 172.21.15.107:0/3571466639 conn(0x7f537006b790 msgr2=0x7f53701e63f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:04.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.481+0000 7f5377015640 1 -- 172.21.15.107:0/3571466639 shutdown_connections 2023-12-04T05:42:04.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:04.481+0000 7f5377015640 1 -- 172.21.15.107:0/3571466639 wait complete. 2023-12-04T05:42:04.502 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-12-04T05:42:05.228 INFO:tasks.workunit.client.0.smithi107.stderr:Updating files: 91% (54/59) Updating files: 93% (55/59) Updating files: 94% (56/59) Updating files: 96% (57/59) Updating files: 98% (58/59) Updating files: 100% (59/59) Updating files: 100% (59/59), done. 2023-12-04T05:42:05.355 INFO:tasks.workunit.client.0.smithi107.stderr:+ trap - EXIT 2023-12-04T05:42:05.355 INFO:tasks.workunit.client.0.smithi107.stderr:+ cd ffsb 2023-12-04T05:42:05.357 INFO:tasks.workunit.client.0.smithi107.stderr:+ ./configure 2023-12-04T05:42:05.674 INFO:tasks.workunit.client.0.smithi107.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2023-12-04T05:42:06.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:42:06.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f33937c5640 1 Processor -- start 2023-12-04T05:42:06.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f33937c5640 1 -- start start 2023-12-04T05:42:06.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f33937c5640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f338c0c8c80 0x7f338c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:06.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f33937c5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f338c06e320 con 0x7f338c0c6b70 2023-12-04T05:42:06.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f33937c5640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f338c0c66e0 con 0x7f338c154290 2023-12-04T05:42:06.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f33937c5640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f338c071c10 con 0x7f338c0c8c80 2023-12-04T05:42:06.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3391d62640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f338c0c6b70 0x7f338c15a710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56840/0 (socket says 172.21.15.107:56840) 2023-12-04T05:42:06.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3391d62640 1 -- 172.21.15.107:0/885156104 learned_addr learned my addr 172.21.15.107:0/885156104 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:42:06.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3390d60640 1 --2- 172.21.15.107:0/885156104 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f338c0c8c80 0x7f338c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:06.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3383fff640 1 -- 172.21.15.107:0/885156104 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1663184819 0 0) 0x7f338c06e320 con 0x7f338c0c6b70 2023-12-04T05:42:06.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3383fff640 1 -- 172.21.15.107:0/885156104 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3378003530 con 0x7f338c0c6b70 2023-12-04T05:42:06.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3383fff640 1 -- 172.21.15.107:0/885156104 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2428224922 0 0) 0x7f3378003530 con 0x7f338c0c6b70 2023-12-04T05:42:06.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3383fff640 1 -- 172.21.15.107:0/885156104 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f338c0c8c80 msgr2=0x7f338c06fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:06.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3383fff640 1 --2- 172.21.15.107:0/885156104 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f338c0c8c80 0x7f338c06fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:06.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3383fff640 1 -- 172.21.15.107:0/885156104 >> v1:172.21.15.195:6789/0 conn(0x7f338c154290 legacy=0x7f338c15a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:06.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3383fff640 1 -- 172.21.15.107:0/885156104 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338c15ae20 con 0x7f338c0c6b70 2023-12-04T05:42:06.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3383fff640 1 -- 172.21.15.107:0/885156104 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f3384001a00 con 0x7f338c0c6b70 2023-12-04T05:42:06.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3383fff640 1 -- 172.21.15.107:0/885156104 >> v1:172.21.15.107:6789/0 conn(0x7f338c0c6b70 legacy=0x7f338c15a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:06.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3383fff640 1 --2- 172.21.15.107:0/885156104 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f33780039b0 0x7f3378003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:06.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3383fff640 1 --2- 172.21.15.107:0/885156104 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f338c0c8c80 0x7f33780047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:06.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3383fff640 1 --2- 172.21.15.107:0/885156104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3378004d10 0x7f3378005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:06.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3383fff640 1 -- 172.21.15.107:0/885156104 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337800bb50 con 0x7f3378004d10 2023-12-04T05:42:06.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3383fff640 1 -- 172.21.15.107:0/885156104 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337800bcc0 con 0x7f33780039b0 2023-12-04T05:42:06.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3383fff640 1 -- 172.21.15.107:0/885156104 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337800be80 con 0x7f338c0c8c80 2023-12-04T05:42:06.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3391d62640 1 --2- 172.21.15.107:0/885156104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3378004d10 0x7f3378005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:06.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3391561640 1 --2- 172.21.15.107:0/885156104 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f33780039b0 0x7f3378003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:06.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3390d60640 1 --2- 172.21.15.107:0/885156104 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f338c0c8c80 0x7f33780047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:06.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3391d62640 1 -- 172.21.15.107:0/885156104 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f338c0c8c80 msgr2=0x7f33780047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:06.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3391d62640 1 --2- 172.21.15.107:0/885156104 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f338c0c8c80 0x7f33780047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:06.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3391d62640 1 -- 172.21.15.107:0/885156104 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f33780039b0 msgr2=0x7f3378003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:06.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3391d62640 1 --2- 172.21.15.107:0/885156104 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f33780039b0 0x7f3378003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:06.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3391d62640 1 -- 172.21.15.107:0/885156104 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3378003530 con 0x7f3378004d10 2023-12-04T05:42:06.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.629+0000 7f3391561640 1 --2- 172.21.15.107:0/885156104 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f33780039b0 0x7f3378003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:42:06.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.633+0000 7f3391d62640 1 --2- 172.21.15.107:0/885156104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3378004d10 0x7f3378005190 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f338400a830 tx=0x7f338401d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e51615ac886f35 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:06.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.633+0000 7f3383fff640 1 -- 172.21.15.107:0/885156104 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33840072d0 con 0x7f3378004d10 2023-12-04T05:42:06.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.633+0000 7f3383fff640 1 -- 172.21.15.107:0/885156104 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33840a8070 con 0x7f3378004d10 2023-12-04T05:42:06.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.633+0000 7f3383fff640 1 -- 172.21.15.107:0/885156104 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3384007690 con 0x7f3378004d10 2023-12-04T05:42:06.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.633+0000 7f33937c5640 1 -- 172.21.15.107:0/885156104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3378004d10 msgr2=0x7f3378005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:06.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.633+0000 7f33937c5640 1 --2- 172.21.15.107:0/885156104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3378004d10 0x7f3378005190 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f338400a830 tx=0x7f338401d340 comp rx=0 tx=0).stop 2023-12-04T05:42:06.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.633+0000 7f3391d62640 1 -- 172.21.15.107:0/885156104 reap_dead start 2023-12-04T05:42:06.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.633+0000 7f33937c5640 1 -- 172.21.15.107:0/885156104 shutdown_connections 2023-12-04T05:42:06.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.633+0000 7f33937c5640 1 -- 172.21.15.107:0/885156104 >> 172.21.15.107:0/885156104 conn(0x7f338c0c7ff0 msgr2=0x7f338c0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:06.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.633+0000 7f33937c5640 1 -- 172.21.15.107:0/885156104 shutdown_connections 2023-12-04T05:42:06.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.633+0000 7f33937c5640 1 -- 172.21.15.107:0/885156104 wait complete. 2023-12-04T05:42:06.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.633+0000 7f33937c5640 1 Processor -- start 2023-12-04T05:42:06.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.633+0000 7f33937c5640 1 -- start start 2023-12-04T05:42:06.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f33937c5640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f338c0c9be0 0x7f338c1593f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:06.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f33937c5640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f338c0c9fc0 0x7f338c159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:06.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f33937c5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f338c1f29a0 0x7f338c1f2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:06.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f33937c5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338c0ca6d0 con 0x7f338c1f29a0 2023-12-04T05:42:06.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f33937c5640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338c1f6b00 con 0x7f338c0c9fc0 2023-12-04T05:42:06.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f33937c5640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f338c1f3360 con 0x7f338c0c9be0 2023-12-04T05:42:06.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f3391d62640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f338c1f29a0 0x7f338c1f2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:06.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f3391d62640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f338c1f29a0 0x7f338c1f2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58712/0 (socket says 172.21.15.107:58712) 2023-12-04T05:42:06.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f3391d62640 1 -- 172.21.15.107:0/1336139442 learned_addr learned my addr 172.21.15.107:0/1336139442 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:06.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f3391d62640 1 -- 172.21.15.107:0/1336139442 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f338c0c9be0 msgr2=0x7f338c1593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:06.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f3391d62640 1 --2- 172.21.15.107:0/1336139442 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f338c0c9be0 0x7f338c1593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:06.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f3391d62640 1 -- 172.21.15.107:0/1336139442 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f338c0c9fc0 msgr2=0x7f338c159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:06.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f3391d62640 1 --2- 172.21.15.107:0/1336139442 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f338c0c9fc0 0x7f338c159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:06.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f3391d62640 1 -- 172.21.15.107:0/1336139442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338c1f5c30 con 0x7f338c1f29a0 2023-12-04T05:42:06.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f3391d62640 1 --2- 172.21.15.107:0/1336139442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f338c1f29a0 0x7f338c1f2e20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f338401dbe0 tx=0x7f3378005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:06.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f3381ffb640 1 -- 172.21.15.107:0/1336139442 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33840ba070 con 0x7f338c1f29a0 2023-12-04T05:42:06.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f3381ffb640 1 -- 172.21.15.107:0/1336139442 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33840011d0 con 0x7f338c1f29a0 2023-12-04T05:42:06.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f33937c5640 1 -- 172.21.15.107:0/1336139442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f338c1f5ef0 con 0x7f338c1f29a0 2023-12-04T05:42:06.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f3381ffb640 1 -- 172.21.15.107:0/1336139442 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3384001340 con 0x7f338c1f29a0 2023-12-04T05:42:06.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f33937c5640 1 -- 172.21.15.107:0/1336139442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f338c1f64b0 con 0x7f338c1f29a0 2023-12-04T05:42:06.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f33937c5640 1 -- 172.21.15.107:0/1336139442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f338c067870 con 0x7f338c1f29a0 2023-12-04T05:42:06.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f3381ffb640 1 -- 172.21.15.107:0/1336139442 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f33840aa030 con 0x7f338c1f29a0 2023-12-04T05:42:06.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f3381ffb640 1 --2- 172.21.15.107:0/1336139442 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f33700668d0 0x7f3370068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:06.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f3391561640 1 --2- 172.21.15.107:0/1336139442 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f33700668d0 0x7f3370068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:06.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f3381ffb640 1 -- 172.21.15.107:0/1336139442 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f338411dac0 con 0x7f338c1f29a0 2023-12-04T05:42:06.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.637+0000 7f3391561640 1 --2- 172.21.15.107:0/1336139442 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f33700668d0 0x7f3370068d90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f337c001620 tx=0x7f337c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:06.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.641+0000 7f3381ffb640 1 -- 172.21.15.107:0/1336139442 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f33840eabf0 con 0x7f338c1f29a0 2023-12-04T05:42:06.698 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether build environment is sane... yes 2023-12-04T05:42:06.735 INFO:tasks.workunit.client.0.smithi107.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2023-12-04T05:42:06.735 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gawk... gawk 2023-12-04T05:42:06.772 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether make sets $(MAKE)... yes 2023-12-04T05:42:06.834 INFO:tasks.workunit.client.0.smithi107.stdout:checking build system type... x86_64-unknown-linux-gnu 2023-12-04T05:42:06.834 INFO:tasks.workunit.client.0.smithi107.stdout:checking host system type... x86_64-unknown-linux-gnu 2023-12-04T05:42:06.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.845+0000 7f33937c5640 1 -- 172.21.15.107:0/1336139442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f338c0c7850 con 0x7f338c1f29a0 2023-12-04T05:42:06.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.845+0000 7f3381ffb640 1 -- 172.21.15.107:0/1336139442 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6563 (secure 0 0 0) 0x7f338411de60 con 0x7f338c1f29a0 2023-12-04T05:42:06.852 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:42:06.852 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:40:28.948661+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":10,"addr":"172.21.15.195:6836/3769196693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":3769196693},{"type":"v1","addr":"172.21.15.195:6836","nonce":3769196693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:42:06.852 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-12-04T05:42:06.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.849+0000 7f33937c5640 1 -- 172.21.15.107:0/1336139442 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f33700668d0 msgr2=0x7f3370068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:06.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.849+0000 7f33937c5640 1 --2- 172.21.15.107:0/1336139442 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f33700668d0 0x7f3370068d90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f337c001620 tx=0x7f337c009040 comp rx=0 tx=0).stop 2023-12-04T05:42:06.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.849+0000 7f33937c5640 1 -- 172.21.15.107:0/1336139442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f338c1f29a0 msgr2=0x7f338c1f2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:06.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.849+0000 7f33937c5640 1 --2- 172.21.15.107:0/1336139442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f338c1f29a0 0x7f338c1f2e20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f338401dbe0 tx=0x7f3378005800 comp rx=0 tx=0).stop 2023-12-04T05:42:06.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.849+0000 7f33937c5640 1 -- 172.21.15.107:0/1336139442 shutdown_connections 2023-12-04T05:42:06.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.849+0000 7f33937c5640 1 --2- 172.21.15.107:0/1336139442 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f33700668d0 0x7f3370068d90 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:06.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.849+0000 7f33937c5640 1 --2- 172.21.15.107:0/1336139442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f338c1f29a0 0x7f338c1f2e20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:06.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.849+0000 7f33937c5640 1 --2- 172.21.15.107:0/1336139442 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f338c0c9fc0 0x7f338c159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:06.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.849+0000 7f33937c5640 1 --2- 172.21.15.107:0/1336139442 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f338c0c9be0 0x7f338c1593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:06.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.849+0000 7f33937c5640 1 -- 172.21.15.107:0/1336139442 >> 172.21.15.107:0/1336139442 conn(0x7f338c0c7ff0 msgr2=0x7f338c164f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:06.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.849+0000 7f33937c5640 1 -- 172.21.15.107:0/1336139442 shutdown_connections 2023-12-04T05:42:06.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:06.849+0000 7f33937c5640 1 -- 172.21.15.107:0/1336139442 wait complete. 2023-12-04T05:42:06.870 INFO:tasks.workunit.client.0.smithi107.stdout:checking for style of include used by make... GNU 2023-12-04T05:42:06.871 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-12-04T05:42:06.888 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc... gcc 2023-12-04T05:42:06.993 INFO:tasks.workunit.client.0.smithi107.stdout:checking for C compiler default output file name... a.out 2023-12-04T05:42:06.996 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether the C compiler works... yes 2023-12-04T05:42:07.009 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether we are cross compiling... no 2023-12-04T05:42:07.084 INFO:tasks.workunit.client.0.smithi107.stdout:checking for suffix of executables... 2023-12-04T05:42:07.164 INFO:tasks.workunit.client.0.smithi107.stdout:checking for suffix of object files... o 2023-12-04T05:42:07.264 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether we are using the GNU C compiler... yes 2023-12-04T05:42:07.798 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether gcc accepts -g... yes 2023-12-04T05:42:07.999 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc option to accept ISO C89... none needed 2023-12-04T05:42:08.873 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:42:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.001+0000 7ff68f469640 1 Processor -- start 2023-12-04T05:42:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.001+0000 7ff68f469640 1 -- start start 2023-12-04T05:42:09.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.005+0000 7ff68f469640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff6880d01c0 0x7ff688071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:09.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.005+0000 7ff68f469640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff688072220 0x7ff6880c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:09.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.005+0000 7ff68f469640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff68815d710 con 0x7ff688154020 2023-12-04T05:42:09.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.005+0000 7ff68f469640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6880ca230 con 0x7ff688072220 2023-12-04T05:42:09.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.005+0000 7ff68f469640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6880c6770 con 0x7ff6880d01c0 2023-12-04T05:42:09.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.005+0000 7ff68d205640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff688154020 0x7ff68815a810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57148/0 (socket says 172.21.15.107:57148) 2023-12-04T05:42:09.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.005+0000 7ff68d205640 1 -- 172.21.15.107:0/1397754440 learned_addr learned my addr 172.21.15.107:0/1397754440 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:42:09.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.005+0000 7ff68da06640 1 --2- 172.21.15.107:0/1397754440 >> v2:172.21.15.107:3300/0 conn(0x7ff688072220 0x7ff6880c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:09.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.005+0000 7ff68ca04640 1 --2- 172.21.15.107:0/1397754440 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff6880d01c0 0x7ff688071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:09.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.005+0000 7ff67ffff640 1 -- 172.21.15.107:0/1397754440 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172250969 0 0) 0x7ff68815d710 con 0x7ff688154020 2023-12-04T05:42:09.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.005+0000 7ff67ffff640 1 -- 172.21.15.107:0/1397754440 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff674003530 con 0x7ff688154020 2023-12-04T05:42:09.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.005+0000 7ff68da06640 1 -- 172.21.15.107:0/1397754440 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff6880d01c0 msgr2=0x7ff688071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:09.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.005+0000 7ff68da06640 1 --2- 172.21.15.107:0/1397754440 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff6880d01c0 0x7ff688071ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:09.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.005+0000 7ff68da06640 1 -- 172.21.15.107:0/1397754440 >> v1:172.21.15.107:6789/0 conn(0x7ff688154020 legacy=0x7ff68815a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:09.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.005+0000 7ff68da06640 1 -- 172.21.15.107:0/1397754440 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff68815d9f0 con 0x7ff688072220 2023-12-04T05:42:09.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.005+0000 7ff68da06640 1 --2- 172.21.15.107:0/1397754440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff688072220 0x7ff6880c4c10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff680009dc0 tx=0x7ff68001f120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c52f100914d2985a server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:09.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff67ffff640 1 -- 172.21.15.107:0/1397754440 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6800af600 con 0x7ff688072220 2023-12-04T05:42:09.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff67ffff640 1 -- 172.21.15.107:0/1397754440 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6800ab050 con 0x7ff688072220 2023-12-04T05:42:09.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff67ffff640 1 -- 172.21.15.107:0/1397754440 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6800af960 con 0x7ff688072220 2023-12-04T05:42:09.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68f469640 1 -- 172.21.15.107:0/1397754440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff688072220 msgr2=0x7ff6880c4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:09.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68f469640 1 --2- 172.21.15.107:0/1397754440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff688072220 0x7ff6880c4c10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff680009dc0 tx=0x7ff68001f120 comp rx=0 tx=0).stop 2023-12-04T05:42:09.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68f469640 1 -- 172.21.15.107:0/1397754440 shutdown_connections 2023-12-04T05:42:09.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68f469640 1 --2- 172.21.15.107:0/1397754440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff688072220 0x7ff6880c4c10 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:09.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68f469640 1 --2- 172.21.15.107:0/1397754440 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff6880d01c0 0x7ff688071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:09.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68f469640 1 -- 172.21.15.107:0/1397754440 >> 172.21.15.107:0/1397754440 conn(0x7ff68806c6c0 msgr2=0x7ff6880cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:09.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68f469640 1 -- 172.21.15.107:0/1397754440 shutdown_connections 2023-12-04T05:42:09.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68f469640 1 -- 172.21.15.107:0/1397754440 wait complete. 2023-12-04T05:42:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68f469640 1 Processor -- start 2023-12-04T05:42:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68f469640 1 -- start start 2023-12-04T05:42:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68f469640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff688072220 0x7ff6881586a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68f469640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6880d01c0 0x7ff6881592d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68f469640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff688154020 0x7ff688159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68f469640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff688159090 con 0x7ff6880d01c0 2023-12-04T05:42:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68f469640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6880cf5b0 con 0x7ff688072220 2023-12-04T05:42:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68f469640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6880cf3e0 con 0x7ff688154020 2023-12-04T05:42:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68ca04640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6880d01c0 0x7ff6881592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68ca04640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6880d01c0 0x7ff6881592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34706/0 (socket says 172.21.15.107:34706) 2023-12-04T05:42:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68ca04640 1 -- 172.21.15.107:0/970890904 learned_addr learned my addr 172.21.15.107:0/970890904 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68d205640 1 --2- 172.21.15.107:0/970890904 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff688072220 0x7ff6881586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:09.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68da06640 1 --2- 172.21.15.107:0/970890904 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff688154020 0x7ff688159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:09.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68ca04640 1 -- 172.21.15.107:0/970890904 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff688154020 msgr2=0x7ff688159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:09.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68ca04640 1 --2- 172.21.15.107:0/970890904 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff688154020 0x7ff688159810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:09.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68ca04640 1 -- 172.21.15.107:0/970890904 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff688072220 msgr2=0x7ff6881586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:09.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68ca04640 1 --2- 172.21.15.107:0/970890904 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff688072220 0x7ff6881586a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:09.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68ca04640 1 -- 172.21.15.107:0/970890904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff680009a70 con 0x7ff6880d01c0 2023-12-04T05:42:09.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68d205640 1 --2- 172.21.15.107:0/970890904 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff688072220 0x7ff6881586a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:42:09.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68ca04640 1 --2- 172.21.15.107:0/970890904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6880d01c0 0x7ff6881592d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff67800d800 tx=0x7ff67800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:09.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff67dffb640 1 -- 172.21.15.107:0/970890904 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67800b6c0 con 0x7ff6880d01c0 2023-12-04T05:42:09.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff67dffb640 1 -- 172.21.15.107:0/970890904 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67800b820 con 0x7ff6880d01c0 2023-12-04T05:42:09.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68f469640 1 -- 172.21.15.107:0/970890904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6880c9660 con 0x7ff6880d01c0 2023-12-04T05:42:09.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff67dffb640 1 -- 172.21.15.107:0/970890904 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67800bb90 con 0x7ff6880d01c0 2023-12-04T05:42:09.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.009+0000 7ff68f469640 1 -- 172.21.15.107:0/970890904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6880c92b0 con 0x7ff6880d01c0 2023-12-04T05:42:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.013+0000 7ff68f469640 1 -- 172.21.15.107:0/970890904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6880cf6f0 con 0x7ff6880d01c0 2023-12-04T05:42:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.013+0000 7ff67dffb640 1 -- 172.21.15.107:0/970890904 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff67800bcf0 con 0x7ff6880d01c0 2023-12-04T05:42:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.013+0000 7ff67dffb640 1 --2- 172.21.15.107:0/970890904 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff66c066900 0x7ff66c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.013+0000 7ff68d205640 1 --2- 172.21.15.107:0/970890904 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff66c066900 0x7ff66c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.013+0000 7ff67dffb640 1 -- 172.21.15.107:0/970890904 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff6780849a0 con 0x7ff6880d01c0 2023-12-04T05:42:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.013+0000 7ff68d205640 1 --2- 172.21.15.107:0/970890904 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff66c066900 0x7ff66c068dc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff6840011f0 tx=0x7ff684009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.017+0000 7ff67dffb640 1 -- 172.21.15.107:0/970890904 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff678051bf0 con 0x7ff6880d01c0 2023-12-04T05:42:09.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.205+0000 7ff68f469640 1 -- 172.21.15.107:0/970890904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff688067870 con 0x7ff6880d01c0 2023-12-04T05:42:09.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.205+0000 7ff67dffb640 1 -- 172.21.15.107:0/970890904 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6563 (secure 0 0 0) 0x7ff678055210 con 0x7ff6880d01c0 2023-12-04T05:42:09.210 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:42:09.210 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:40:28.948661+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":10,"addr":"172.21.15.195:6836/3769196693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":3769196693},{"type":"v1","addr":"172.21.15.195:6836","nonce":3769196693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:42:09.210 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-12-04T05:42:09.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.205+0000 7ff68f469640 1 -- 172.21.15.107:0/970890904 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff66c066900 msgr2=0x7ff66c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:09.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.205+0000 7ff68f469640 1 --2- 172.21.15.107:0/970890904 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff66c066900 0x7ff66c068dc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff6840011f0 tx=0x7ff684009040 comp rx=0 tx=0).stop 2023-12-04T05:42:09.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.205+0000 7ff68f469640 1 -- 172.21.15.107:0/970890904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6880d01c0 msgr2=0x7ff6881592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:09.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.205+0000 7ff68f469640 1 --2- 172.21.15.107:0/970890904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6880d01c0 0x7ff6881592d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff67800d800 tx=0x7ff67800dcc0 comp rx=0 tx=0).stop 2023-12-04T05:42:09.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.205+0000 7ff68f469640 1 -- 172.21.15.107:0/970890904 shutdown_connections 2023-12-04T05:42:09.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.205+0000 7ff68f469640 1 --2- 172.21.15.107:0/970890904 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff66c066900 0x7ff66c068dc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:09.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.205+0000 7ff68f469640 1 --2- 172.21.15.107:0/970890904 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff688154020 0x7ff688159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:09.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.205+0000 7ff68f469640 1 --2- 172.21.15.107:0/970890904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6880d01c0 0x7ff6881592d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:09.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.205+0000 7ff68f469640 1 --2- 172.21.15.107:0/970890904 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff688072220 0x7ff6881586a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:09.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.205+0000 7ff68f469640 1 -- 172.21.15.107:0/970890904 >> 172.21.15.107:0/970890904 conn(0x7ff68806c6c0 msgr2=0x7ff688152e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:09.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.205+0000 7ff68f469640 1 -- 172.21.15.107:0/970890904 shutdown_connections 2023-12-04T05:42:09.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:09.205+0000 7ff68f469640 1 -- 172.21.15.107:0/970890904 wait complete. 2023-12-04T05:42:09.229 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-12-04T05:42:10.489 INFO:tasks.workunit.client.0.smithi107.stdout:checking dependency style of gcc... gcc3 2023-12-04T05:42:10.704 INFO:tasks.workunit.client.0.smithi107.stdout:checking how to run the C preprocessor... gcc -E 2023-12-04T05:42:11.024 INFO:tasks.workunit.client.0.smithi107.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2023-12-04T05:42:11.027 INFO:tasks.workunit.client.0.smithi107.stdout:checking for egrep... /usr/bin/grep -E 2023-12-04T05:42:11.052 INFO:tasks.workunit.client.0.smithi107.stdout:checking for AIX... no 2023-12-04T05:42:11.072 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc... (cached) gcc 2023-12-04T05:42:11.090 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether we are using the GNU C compiler... (cached) yes 2023-12-04T05:42:11.091 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether gcc accepts -g... (cached) yes 2023-12-04T05:42:11.093 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2023-12-04T05:42:11.093 INFO:tasks.workunit.client.0.smithi107.stdout:checking dependency style of gcc... (cached) gcc3 2023-12-04T05:42:11.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:42:11.271 INFO:tasks.workunit.client.0.smithi107.stdout:checking for main in -lm... yes 2023-12-04T05:42:11.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f7dde640 1 Processor -- start 2023-12-04T05:42:11.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f7dde640 1 -- start start 2023-12-04T05:42:11.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f7dde640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f72f006e1f0 0x7f72f006e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:11.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f7dde640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f0152360 con 0x7f72f0151f80 2023-12-04T05:42:11.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f7dde640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f0071460 con 0x7f72f0071030 2023-12-04T05:42:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f7dde640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f006acf0 con 0x7f72f006e1f0 2023-12-04T05:42:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f5b7a640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f72f0151f80 0x7f72f006c2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57150/0 (socket says 172.21.15.107:57150) 2023-12-04T05:42:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f5b7a640 1 -- 172.21.15.107:0/3206859333 learned_addr learned my addr 172.21.15.107:0/3206859333 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:42:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f4b78640 1 -- 172.21.15.107:0/3206859333 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584089464 0 0) 0x7f72f0152360 con 0x7f72f0151f80 2023-12-04T05:42:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f4b78640 1 -- 172.21.15.107:0/3206859333 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72dc003530 con 0x7f72f0151f80 2023-12-04T05:42:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f637b640 1 --2- 172.21.15.107:0/3206859333 >> v2:172.21.15.195:3300/0 conn(0x7f72f006e1f0 0x7f72f006e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f4b78640 1 -- 172.21.15.107:0/3206859333 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 849815431 0 0) 0x7f72dc003530 con 0x7f72f0151f80 2023-12-04T05:42:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f4b78640 1 -- 172.21.15.107:0/3206859333 >> v2:172.21.15.195:3300/0 conn(0x7f72f006e1f0 msgr2=0x7f72f006e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f4b78640 1 --2- 172.21.15.107:0/3206859333 >> v2:172.21.15.195:3300/0 conn(0x7f72f006e1f0 0x7f72f006e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f4b78640 1 -- 172.21.15.107:0/3206859333 >> v1:172.21.15.195:6789/0 conn(0x7f72f0071030 legacy=0x7f72f00c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f4b78640 1 -- 172.21.15.107:0/3206859333 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f0157c90 con 0x7f72f0151f80 2023-12-04T05:42:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f4b78640 1 -- 172.21.15.107:0/3206859333 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f72e8001a00 con 0x7f72f0151f80 2023-12-04T05:42:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f4b78640 1 -- 172.21.15.107:0/3206859333 >> v1:172.21.15.107:6789/0 conn(0x7f72f0151f80 legacy=0x7f72f006c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f4b78640 1 --2- 172.21.15.107:0/3206859333 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f72dc003980 0x7f72dc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f4b78640 1 --2- 172.21.15.107:0/3206859333 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f72dc0047d0 0x7f72dc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f4b78640 1 --2- 172.21.15.107:0/3206859333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc005190 0x7f72dc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f4b78640 1 -- 172.21.15.107:0/3206859333 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f006acf0 con 0x7f72dc005190 2023-12-04T05:42:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f4b78640 1 -- 172.21.15.107:0/3206859333 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc00c0c0 con 0x7f72dc003980 2023-12-04T05:42:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f4b78640 1 -- 172.21.15.107:0/3206859333 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc00c280 con 0x7f72dc0047d0 2023-12-04T05:42:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f637b640 1 --2- 172.21.15.107:0/3206859333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc005190 0x7f72dc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f5b7a640 1 --2- 172.21.15.107:0/3206859333 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f72dc003980 0x7f72dc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f5379640 1 --2- 172.21.15.107:0/3206859333 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f72dc0047d0 0x7f72dc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f637b640 1 -- 172.21.15.107:0/3206859333 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f72dc0047d0 msgr2=0x7f72dc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f637b640 1 --2- 172.21.15.107:0/3206859333 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f72dc0047d0 0x7f72dc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f637b640 1 -- 172.21.15.107:0/3206859333 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f72dc003980 msgr2=0x7f72dc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f637b640 1 --2- 172.21.15.107:0/3206859333 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f72dc003980 0x7f72dc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f637b640 1 -- 172.21.15.107:0/3206859333 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72dc003530 con 0x7f72dc005190 2023-12-04T05:42:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.357+0000 7f72f5379640 1 --2- 172.21.15.107:0/3206859333 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f72dc0047d0 0x7f72dc004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:42:11.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.361+0000 7f72f637b640 1 -- 172.21.15.107:0/3206859333 reap_dead start 2023-12-04T05:42:11.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.361+0000 7f72f637b640 1 --2- 172.21.15.107:0/3206859333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc005190 0x7f72dc00b990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f72ec00ba30 tx=0x7f72ec01c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0874f50a4b6d737 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:11.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.361+0000 7f72f4b78640 1 -- 172.21.15.107:0/3206859333 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec004220 con 0x7f72dc005190 2023-12-04T05:42:11.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.361+0000 7f72f4b78640 1 -- 172.21.15.107:0/3206859333 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ec0043c0 con 0x7f72dc005190 2023-12-04T05:42:11.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.361+0000 7f72f4b78640 1 -- 172.21.15.107:0/3206859333 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec003860 con 0x7f72dc005190 2023-12-04T05:42:11.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.361+0000 7f72f7dde640 1 -- 172.21.15.107:0/3206859333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc005190 msgr2=0x7f72dc00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:11.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.361+0000 7f72f7dde640 1 --2- 172.21.15.107:0/3206859333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc005190 0x7f72dc00b990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f72ec00ba30 tx=0x7f72ec01c6c0 comp rx=0 tx=0).stop 2023-12-04T05:42:11.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.361+0000 7f72f7dde640 1 -- 172.21.15.107:0/3206859333 shutdown_connections 2023-12-04T05:42:11.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.361+0000 7f72f7dde640 1 --2- 172.21.15.107:0/3206859333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc005190 0x7f72dc00b990 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:11.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.361+0000 7f72f7dde640 1 -- 172.21.15.107:0/3206859333 >> 172.21.15.107:0/3206859333 conn(0x7f72f00cc490 msgr2=0x7f72f00cc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:11.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.361+0000 7f72f7dde640 1 -- 172.21.15.107:0/3206859333 shutdown_connections 2023-12-04T05:42:11.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.361+0000 7f72f7dde640 1 -- 172.21.15.107:0/3206859333 wait complete. 2023-12-04T05:42:11.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.361+0000 7f72f7dde640 1 Processor -- start 2023-12-04T05:42:11.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.361+0000 7f72f7dde640 1 -- start start 2023-12-04T05:42:11.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72f7dde640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc005190 0x7f72f00c91d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:11.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72f7dde640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f72f0153520 0x7f72f0149da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:11.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72f7dde640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f72f014a2e0 0x7f72f0153c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:11.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72f7dde640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f0158f20 con 0x7f72dc005190 2023-12-04T05:42:11.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72f7dde640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f015be40 con 0x7f72f0153520 2023-12-04T05:42:11.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72f7dde640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f014a790 con 0x7f72f014a2e0 2023-12-04T05:42:11.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72f5b7a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc005190 0x7f72f00c91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:11.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72f5b7a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc005190 0x7f72f00c91d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34730/0 (socket says 172.21.15.107:34730) 2023-12-04T05:42:11.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72f5b7a640 1 -- 172.21.15.107:0/1123548551 learned_addr learned my addr 172.21.15.107:0/1123548551 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:11.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72f5379640 1 --2- 172.21.15.107:0/1123548551 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f72f0153520 0x7f72f0149da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:11.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72f637b640 1 --2- 172.21.15.107:0/1123548551 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f72f014a2e0 0x7f72f0153c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:11.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72f5b7a640 1 -- 172.21.15.107:0/1123548551 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f72f014a2e0 msgr2=0x7f72f0153c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:11.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72f5b7a640 1 --2- 172.21.15.107:0/1123548551 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f72f014a2e0 0x7f72f0153c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:11.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72f5b7a640 1 -- 172.21.15.107:0/1123548551 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f72f0153520 msgr2=0x7f72f0149da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:11.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72f5b7a640 1 --2- 172.21.15.107:0/1123548551 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f72f0153520 0x7f72f0149da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:11.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72f5b7a640 1 -- 172.21.15.107:0/1123548551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f01ecf90 con 0x7f72dc005190 2023-12-04T05:42:11.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72f5b7a640 1 --2- 172.21.15.107:0/1123548551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc005190 0x7f72f00c91d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f72e80079d0 tx=0x7f72e800a6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:11.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72e67fc640 1 -- 172.21.15.107:0/1123548551 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e80133f0 con 0x7f72dc005190 2023-12-04T05:42:11.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72e67fc640 1 -- 172.21.15.107:0/1123548551 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e8013550 con 0x7f72dc005190 2023-12-04T05:42:11.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72f7dde640 1 -- 172.21.15.107:0/1123548551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f01ed270 con 0x7f72dc005190 2023-12-04T05:42:11.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72e67fc640 1 -- 172.21.15.107:0/1123548551 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e8013900 con 0x7f72dc005190 2023-12-04T05:42:11.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72f7dde640 1 -- 172.21.15.107:0/1123548551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f01ed830 con 0x7f72dc005190 2023-12-04T05:42:11.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.365+0000 7f72f7dde640 1 -- 172.21.15.107:0/1123548551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f0067ec0 con 0x7f72dc005190 2023-12-04T05:42:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.369+0000 7f72e67fc640 1 -- 172.21.15.107:0/1123548551 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f72e8013a60 con 0x7f72dc005190 2023-12-04T05:42:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.369+0000 7f72e67fc640 1 --2- 172.21.15.107:0/1123548551 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f72d4066900 0x7f72d4068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.369+0000 7f72e67fc640 1 -- 172.21.15.107:0/1123548551 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f72e8084f80 con 0x7f72dc005190 2023-12-04T05:42:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.369+0000 7f72f5379640 1 --2- 172.21.15.107:0/1123548551 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f72d4066900 0x7f72d4068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.369+0000 7f72f5379640 1 --2- 172.21.15.107:0/1123548551 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f72d4066900 0x7f72d4068dc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f72e0007fd0 tx=0x7f72e0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:11.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.373+0000 7f72e67fc640 1 -- 172.21.15.107:0/1123548551 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f72e80b6280 con 0x7f72dc005190 2023-12-04T05:42:11.444 INFO:tasks.workunit.client.0.smithi107.stdout:checking for main in -lpthread... yes 2023-12-04T05:42:11.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.565+0000 7f72f7dde640 1 -- 172.21.15.107:0/1123548551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72f0151610 con 0x7f72dc005190 2023-12-04T05:42:11.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.565+0000 7f72e67fc640 1 -- 172.21.15.107:0/1123548551 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6563 (secure 0 0 0) 0x7f72e80520b0 con 0x7f72dc005190 2023-12-04T05:42:11.572 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:42:11.572 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:40:28.948661+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":10,"addr":"172.21.15.195:6836/3769196693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":3769196693},{"type":"v1","addr":"172.21.15.195:6836","nonce":3769196693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:42:11.572 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-12-04T05:42:11.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.569+0000 7f72f7dde640 1 -- 172.21.15.107:0/1123548551 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f72d4066900 msgr2=0x7f72d4068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:11.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.569+0000 7f72f7dde640 1 --2- 172.21.15.107:0/1123548551 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f72d4066900 0x7f72d4068dc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f72e0007fd0 tx=0x7f72e0009040 comp rx=0 tx=0).stop 2023-12-04T05:42:11.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.569+0000 7f72f7dde640 1 -- 172.21.15.107:0/1123548551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc005190 msgr2=0x7f72f00c91d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:11.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.569+0000 7f72f7dde640 1 --2- 172.21.15.107:0/1123548551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc005190 0x7f72f00c91d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f72e80079d0 tx=0x7f72e800a6e0 comp rx=0 tx=0).stop 2023-12-04T05:42:11.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.569+0000 7f72f7dde640 1 -- 172.21.15.107:0/1123548551 shutdown_connections 2023-12-04T05:42:11.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.569+0000 7f72f7dde640 1 --2- 172.21.15.107:0/1123548551 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f72d4066900 0x7f72d4068dc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:11.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.569+0000 7f72f7dde640 1 --2- 172.21.15.107:0/1123548551 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f72f014a2e0 0x7f72f0153c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:11.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.569+0000 7f72f7dde640 1 --2- 172.21.15.107:0/1123548551 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f72f0153520 0x7f72f0149da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:11.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.569+0000 7f72f7dde640 1 --2- 172.21.15.107:0/1123548551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72dc005190 0x7f72f00c91d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:11.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.569+0000 7f72f7dde640 1 -- 172.21.15.107:0/1123548551 >> 172.21.15.107:0/1123548551 conn(0x7f72f00cc490 msgr2=0x7f72f006b740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:11.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.569+0000 7f72f7dde640 1 -- 172.21.15.107:0/1123548551 shutdown_connections 2023-12-04T05:42:11.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:11.569+0000 7f72f7dde640 1 -- 172.21.15.107:0/1123548551 wait complete. 2023-12-04T05:42:11.591 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-12-04T05:42:13.439 INFO:tasks.workunit.client.0.smithi107.stdout:checking for ANSI C header files... yes 2023-12-04T05:42:13.578 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2023-12-04T05:42:13.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:42:13.714 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/types.h... yes 2023-12-04T05:42:13.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.708+0000 7f0f1fb5d640 1 Processor -- start 2023-12-04T05:42:13.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.708+0000 7f0f1fb5d640 1 -- start start 2023-12-04T05:42:13.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.708+0000 7f0f1fb5d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0f1815d6b0 0x7f0f1815dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:13.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.708+0000 7f0f1fb5d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f18154450 con 0x7f0f18154070 2023-12-04T05:42:13.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.708+0000 7f0f1fb5d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f18159ed0 con 0x7f0f180724e0 2023-12-04T05:42:13.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.708+0000 7f0f1fb5d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f18159ce0 con 0x7f0f1815d6b0 2023-12-04T05:42:13.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.708+0000 7f0f1e0fa640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0f1815d6b0 0x7f0f1815dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:13.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.708+0000 7f0f1d8f9640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0f18154070 0x7f0f180cd420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57158/0 (socket says 172.21.15.107:57158) 2023-12-04T05:42:13.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.708+0000 7f0f1d8f9640 1 -- 172.21.15.107:0/3644581068 learned_addr learned my addr 172.21.15.107:0/3644581068 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:42:13.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.708+0000 7f0f1e0fa640 1 -- 172.21.15.107:0/3644581068 >> v1:172.21.15.195:6789/0 conn(0x7f0f180724e0 legacy=0x7f0f18072950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:13.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.708+0000 7f0f1e0fa640 1 -- 172.21.15.107:0/3644581068 >> v1:172.21.15.107:6789/0 conn(0x7f0f18154070 legacy=0x7f0f180cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:13.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.708+0000 7f0f1e0fa640 1 -- 172.21.15.107:0/3644581068 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1815a170 con 0x7f0f1815d6b0 2023-12-04T05:42:13.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.708+0000 7f0f1c8f7640 1 -- 172.21.15.107:0/3644581068 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2419592962 0 0) 0x7f0f14001120 con 0x7f0f18154070 2023-12-04T05:42:13.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.712+0000 7f0f1e0fa640 1 --2- 172.21.15.107:0/3644581068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f1815d6b0 0x7f0f1815dac0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0f10009d90 tx=0x7f0f1001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ba38e1f2f01919c server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:13.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.712+0000 7f0f1c8f7640 1 -- 172.21.15.107:0/3644581068 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f100a58e0 con 0x7f0f1815d6b0 2023-12-04T05:42:13.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.712+0000 7f0f1c8f7640 1 -- 172.21.15.107:0/3644581068 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f100a5a40 con 0x7f0f1815d6b0 2023-12-04T05:42:13.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.712+0000 7f0f1c8f7640 1 -- 172.21.15.107:0/3644581068 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f100a5dd0 con 0x7f0f1815d6b0 2023-12-04T05:42:13.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.712+0000 7f0f1fb5d640 1 -- 172.21.15.107:0/3644581068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f1815d6b0 msgr2=0x7f0f1815dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:13.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.712+0000 7f0f1fb5d640 1 --2- 172.21.15.107:0/3644581068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f1815d6b0 0x7f0f1815dac0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0f10009d90 tx=0x7f0f1001c910 comp rx=0 tx=0).stop 2023-12-04T05:42:13.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.712+0000 7f0f1fb5d640 1 -- 172.21.15.107:0/3644581068 shutdown_connections 2023-12-04T05:42:13.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.712+0000 7f0f1fb5d640 1 --2- 172.21.15.107:0/3644581068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f1815d6b0 0x7f0f1815dac0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:13.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.712+0000 7f0f1fb5d640 1 -- 172.21.15.107:0/3644581068 >> 172.21.15.107:0/3644581068 conn(0x7f0f180663a0 msgr2=0x7f0f180667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:13.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.712+0000 7f0f1fb5d640 1 -- 172.21.15.107:0/3644581068 shutdown_connections 2023-12-04T05:42:13.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.712+0000 7f0f1fb5d640 1 -- 172.21.15.107:0/3644581068 wait complete. 2023-12-04T05:42:13.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1fb5d640 1 Processor -- start 2023-12-04T05:42:13.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1fb5d640 1 -- start start 2023-12-04T05:42:13.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1fb5d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f180724e0 0x7f0f181545f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:13.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1fb5d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0f18154070 0x7f0f18156aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:13.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1fb5d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0f1815d6b0 0x7f0f18156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:13.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1fb5d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f181522d0 con 0x7f0f180724e0 2023-12-04T05:42:13.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1fb5d640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1815e7d0 con 0x7f0f1815d6b0 2023-12-04T05:42:13.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1fb5d640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f18151930 con 0x7f0f18154070 2023-12-04T05:42:13.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1d8f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f180724e0 0x7f0f181545f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:13.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1d8f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f180724e0 0x7f0f181545f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34740/0 (socket says 172.21.15.107:34740) 2023-12-04T05:42:13.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1d8f9640 1 -- 172.21.15.107:0/3379245469 learned_addr learned my addr 172.21.15.107:0/3379245469 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:13.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1e0fa640 1 --2- 172.21.15.107:0/3379245469 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0f1815d6b0 0x7f0f18156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:13.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1d0f8640 1 --2- 172.21.15.107:0/3379245469 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0f18154070 0x7f0f18156aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:13.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1d8f9640 1 -- 172.21.15.107:0/3379245469 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0f18154070 msgr2=0x7f0f18156aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:13.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1d8f9640 1 --2- 172.21.15.107:0/3379245469 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0f18154070 0x7f0f18156aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:13.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1d8f9640 1 -- 172.21.15.107:0/3379245469 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0f1815d6b0 msgr2=0x7f0f18156fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:13.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1d8f9640 1 --2- 172.21.15.107:0/3379245469 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0f1815d6b0 0x7f0f18156fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:13.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1d8f9640 1 -- 172.21.15.107:0/3379245469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f10009a70 con 0x7f0f180724e0 2023-12-04T05:42:13.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1e0fa640 1 --2- 172.21.15.107:0/3379245469 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0f1815d6b0 0x7f0f18156fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:42:13.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1d8f9640 1 --2- 172.21.15.107:0/3379245469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f180724e0 0x7f0f181545f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0f1400c750 tx=0x7f0f1400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:13.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f0e7fc640 1 -- 172.21.15.107:0/3379245469 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f14005ac0 con 0x7f0f180724e0 2023-12-04T05:42:13.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f0e7fc640 1 -- 172.21.15.107:0/3379245469 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f14005c20 con 0x7f0f180724e0 2023-12-04T05:42:13.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1fb5d640 1 -- 172.21.15.107:0/3379245469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f18151c10 con 0x7f0f180724e0 2023-12-04T05:42:13.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f0e7fc640 1 -- 172.21.15.107:0/3379245469 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f14007070 con 0x7f0f180724e0 2023-12-04T05:42:13.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1fb5d640 1 -- 172.21.15.107:0/3379245469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f181526f0 con 0x7f0f180724e0 2023-12-04T05:42:13.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f1fb5d640 1 -- 172.21.15.107:0/3379245469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f18071e90 con 0x7f0f180724e0 2023-12-04T05:42:13.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.716+0000 7f0f0e7fc640 1 -- 172.21.15.107:0/3379245469 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f0f140071d0 con 0x7f0f180724e0 2023-12-04T05:42:13.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.720+0000 7f0f0e7fc640 1 --2- 172.21.15.107:0/3379245469 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0efc0668d0 0x7f0efc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:13.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.720+0000 7f0f0e7fc640 1 -- 172.21.15.107:0/3379245469 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0f140838d0 con 0x7f0f180724e0 2023-12-04T05:42:13.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.720+0000 7f0f1d0f8640 1 --2- 172.21.15.107:0/3379245469 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0efc0668d0 0x7f0efc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:13.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.720+0000 7f0f1d0f8640 1 --2- 172.21.15.107:0/3379245469 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0efc0668d0 0x7f0efc068d90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0f1806c0a0 tx=0x7f0f08009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:13.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.724+0000 7f0f0e7fc640 1 -- 172.21.15.107:0/3379245469 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0f18071e90 con 0x7f0f180724e0 2023-12-04T05:42:13.860 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/stat.h... yes 2023-12-04T05:42:13.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.928+0000 7f0f1fb5d640 1 -- 172.21.15.107:0/3379245469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f18154450 con 0x7f0f180724e0 2023-12-04T05:42:13.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.928+0000 7f0f0e7fc640 1 -- 172.21.15.107:0/3379245469 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6563 (secure 0 0 0) 0x7f0f18154450 con 0x7f0f180724e0 2023-12-04T05:42:13.935 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:42:13.935 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:40:28.948661+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":10,"addr":"172.21.15.195:6836/3769196693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":3769196693},{"type":"v1","addr":"172.21.15.195:6836","nonce":3769196693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:42:13.936 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-12-04T05:42:13.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.932+0000 7f0f1fb5d640 1 -- 172.21.15.107:0/3379245469 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0efc0668d0 msgr2=0x7f0efc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:13.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.932+0000 7f0f1fb5d640 1 --2- 172.21.15.107:0/3379245469 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0efc0668d0 0x7f0efc068d90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0f1806c0a0 tx=0x7f0f08009040 comp rx=0 tx=0).stop 2023-12-04T05:42:13.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.932+0000 7f0f1fb5d640 1 -- 172.21.15.107:0/3379245469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f180724e0 msgr2=0x7f0f181545f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:13.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.932+0000 7f0f1fb5d640 1 --2- 172.21.15.107:0/3379245469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f180724e0 0x7f0f181545f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0f1400c750 tx=0x7f0f1400cc10 comp rx=0 tx=0).stop 2023-12-04T05:42:13.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.932+0000 7f0f1fb5d640 1 -- 172.21.15.107:0/3379245469 shutdown_connections 2023-12-04T05:42:13.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.932+0000 7f0f1fb5d640 1 --2- 172.21.15.107:0/3379245469 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0efc0668d0 0x7f0efc068d90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:13.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.932+0000 7f0f1fb5d640 1 --2- 172.21.15.107:0/3379245469 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0f1815d6b0 0x7f0f18156fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:13.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.932+0000 7f0f1fb5d640 1 --2- 172.21.15.107:0/3379245469 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0f18154070 0x7f0f18156aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:13.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.932+0000 7f0f1fb5d640 1 --2- 172.21.15.107:0/3379245469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f180724e0 0x7f0f181545f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:13.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.932+0000 7f0f1fb5d640 1 -- 172.21.15.107:0/3379245469 >> 172.21.15.107:0/3379245469 conn(0x7f0f180663a0 msgr2=0x7f0f1815cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:13.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.932+0000 7f0f1fb5d640 1 -- 172.21.15.107:0/3379245469 shutdown_connections 2023-12-04T05:42:13.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:13.932+0000 7f0f1fb5d640 1 -- 172.21.15.107:0/3379245469 wait complete. 2023-12-04T05:42:13.955 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-12-04T05:42:14.005 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stdlib.h... yes 2023-12-04T05:42:14.160 INFO:tasks.workunit.client.0.smithi107.stdout:checking for string.h... yes 2023-12-04T05:42:14.315 INFO:tasks.workunit.client.0.smithi107.stdout:checking for memory.h... yes 2023-12-04T05:42:14.451 INFO:tasks.workunit.client.0.smithi107.stdout:checking for strings.h... yes 2023-12-04T05:42:15.246 INFO:tasks.workunit.client.0.smithi107.stdout:checking for inttypes.h... yes 2023-12-04T05:42:15.420 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stdint.h... yes 2023-12-04T05:42:15.580 INFO:tasks.workunit.client.0.smithi107.stdout:checking for unistd.h... yes 2023-12-04T05:42:15.748 INFO:tasks.workunit.client.0.smithi107.stdout:checking pthread.h usability... yes 2023-12-04T05:42:15.830 INFO:tasks.workunit.client.0.smithi107.stdout:checking pthread.h presence... yes 2023-12-04T05:42:15.832 INFO:tasks.workunit.client.0.smithi107.stdout:checking for pthread.h... yes 2023-12-04T05:42:15.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:42:15.997 INFO:tasks.workunit.client.0.smithi107.stdout:checking fcntl.h usability... yes 2023-12-04T05:42:16.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f0885e1f640 1 Processor -- start 2023-12-04T05:42:16.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f0885e1f640 1 -- start start 2023-12-04T05:42:16.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f0885e1f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f08800d01c0 0x7f0880071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:16.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f0885e1f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f088015d710 con 0x7f0880154020 2023-12-04T05:42:16.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f0885e1f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f088015d9f0 con 0x7f0880072220 2023-12-04T05:42:16.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f0885e1f640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08800ca230 con 0x7f08800d01c0 2023-12-04T05:42:16.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f087f7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0880154020 0x7f088015a810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57172/0 (socket says 172.21.15.107:57172) 2023-12-04T05:42:16.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f087f7fe640 1 -- 172.21.15.107:0/623447114 learned_addr learned my addr 172.21.15.107:0/623447114 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:42:16.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f087effd640 1 -- 172.21.15.107:0/623447114 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812888836 0 0) 0x7f088015d710 con 0x7f0880154020 2023-12-04T05:42:16.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f0877fff640 1 --2- 172.21.15.107:0/623447114 >> v2:172.21.15.195:3300/0 conn(0x7f08800d01c0 0x7f0880071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:16.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f087effd640 1 -- 172.21.15.107:0/623447114 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f086c003530 con 0x7f0880154020 2023-12-04T05:42:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f087effd640 1 -- 172.21.15.107:0/623447114 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 796393799 0 0) 0x7f086c003530 con 0x7f0880154020 2023-12-04T05:42:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f087effd640 1 -- 172.21.15.107:0/623447114 >> v2:172.21.15.195:3300/0 conn(0x7f08800d01c0 msgr2=0x7f0880071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f087effd640 1 --2- 172.21.15.107:0/623447114 >> v2:172.21.15.195:3300/0 conn(0x7f08800d01c0 0x7f0880071ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f087effd640 1 -- 172.21.15.107:0/623447114 >> v1:172.21.15.195:6789/0 conn(0x7f0880072220 legacy=0x7f08800c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f087effd640 1 -- 172.21.15.107:0/623447114 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088015dcd0 con 0x7f0880154020 2023-12-04T05:42:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f087effd640 1 -- 172.21.15.107:0/623447114 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f0870001680 con 0x7f0880154020 2023-12-04T05:42:16.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f087effd640 1 -- 172.21.15.107:0/623447114 >> v1:172.21.15.107:6789/0 conn(0x7f0880154020 legacy=0x7f088015a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:16.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f087effd640 1 --2- 172.21.15.107:0/623447114 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f086c003980 0x7f086c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:16.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f087effd640 1 --2- 172.21.15.107:0/623447114 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f086c0047d0 0x7f086c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:16.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f087effd640 1 --2- 172.21.15.107:0/623447114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f086c005190 0x7f086c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:16.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f087effd640 1 -- 172.21.15.107:0/623447114 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086c00c090 con 0x7f086c005190 2023-12-04T05:42:16.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f087effd640 1 -- 172.21.15.107:0/623447114 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086c00c200 con 0x7f086c003980 2023-12-04T05:42:16.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f087effd640 1 -- 172.21.15.107:0/623447114 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f086c00c3c0 con 0x7f086c0047d0 2023-12-04T05:42:16.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.068+0000 7f087ffff640 1 --2- 172.21.15.107:0/623447114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f086c005190 0x7f086c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:16.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.072+0000 7f087f7fe640 1 --2- 172.21.15.107:0/623447114 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f086c003980 0x7f086c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:16.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.072+0000 7f0877fff640 1 --2- 172.21.15.107:0/623447114 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f086c0047d0 0x7f086c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:16.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.072+0000 7f087ffff640 1 -- 172.21.15.107:0/623447114 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f086c0047d0 msgr2=0x7f086c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:16.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.072+0000 7f087ffff640 1 --2- 172.21.15.107:0/623447114 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f086c0047d0 0x7f086c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:16.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.072+0000 7f087ffff640 1 -- 172.21.15.107:0/623447114 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f086c003980 msgr2=0x7f086c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:16.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.072+0000 7f087ffff640 1 --2- 172.21.15.107:0/623447114 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f086c003980 0x7f086c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:16.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.072+0000 7f087ffff640 1 -- 172.21.15.107:0/623447114 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c003530 con 0x7f086c005190 2023-12-04T05:42:16.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.072+0000 7f087ffff640 1 -- 172.21.15.107:0/623447114 reap_dead start 2023-12-04T05:42:16.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.072+0000 7f087ffff640 1 --2- 172.21.15.107:0/623447114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f086c005190 0x7f086c00b990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0878001e70 tx=0x7f087801c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c479b7ce94412737 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:16.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.072+0000 7f087effd640 1 -- 172.21.15.107:0/623447114 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0878003b80 con 0x7f086c005190 2023-12-04T05:42:16.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.072+0000 7f087effd640 1 -- 172.21.15.107:0/623447114 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0878003d20 con 0x7f086c005190 2023-12-04T05:42:16.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.072+0000 7f087effd640 1 -- 172.21.15.107:0/623447114 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0878005260 con 0x7f086c005190 2023-12-04T05:42:16.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.072+0000 7f0885e1f640 1 -- 172.21.15.107:0/623447114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f086c005190 msgr2=0x7f086c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:16.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.072+0000 7f0885e1f640 1 --2- 172.21.15.107:0/623447114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f086c005190 0x7f086c00b990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0878001e70 tx=0x7f087801c760 comp rx=0 tx=0).stop 2023-12-04T05:42:16.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.072+0000 7f0885e1f640 1 -- 172.21.15.107:0/623447114 shutdown_connections 2023-12-04T05:42:16.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.072+0000 7f0885e1f640 1 --2- 172.21.15.107:0/623447114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f086c005190 0x7f086c00b990 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:16.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.072+0000 7f0885e1f640 1 -- 172.21.15.107:0/623447114 >> 172.21.15.107:0/623447114 conn(0x7f088006c6c0 msgr2=0x7f08800cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:16.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.072+0000 7f0885e1f640 1 -- 172.21.15.107:0/623447114 shutdown_connections 2023-12-04T05:42:16.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.072+0000 7f0885e1f640 1 -- 172.21.15.107:0/623447114 wait complete. 2023-12-04T05:42:16.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f0885e1f640 1 Processor -- start 2023-12-04T05:42:16.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f0885e1f640 1 -- start start 2023-12-04T05:42:16.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f0885e1f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f086c005190 0x7f08800cab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:16.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f0885e1f640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08800cb040 0x7f08800ce820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:16.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f0885e1f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f088015af50 0x7f08800ced60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:16.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f0885e1f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08801ef650 con 0x7f086c005190 2023-12-04T05:42:16.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f0885e1f640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08800c8dd0 con 0x7f08800cb040 2023-12-04T05:42:16.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f0885e1f640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08800c81c0 con 0x7f088015af50 2023-12-04T05:42:16.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f087f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f086c005190 0x7f08800cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:16.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f087f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f086c005190 0x7f08800cab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34752/0 (socket says 172.21.15.107:34752) 2023-12-04T05:42:16.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f087f7fe640 1 -- 172.21.15.107:0/1433460367 learned_addr learned my addr 172.21.15.107:0/1433460367 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:16.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f0877fff640 1 --2- 172.21.15.107:0/1433460367 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08800cb040 0x7f08800ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:16.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f087ffff640 1 --2- 172.21.15.107:0/1433460367 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f088015af50 0x7f08800ced60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:16.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f087f7fe640 1 -- 172.21.15.107:0/1433460367 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f088015af50 msgr2=0x7f08800ced60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:16.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f087f7fe640 1 --2- 172.21.15.107:0/1433460367 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f088015af50 0x7f08800ced60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:16.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f087f7fe640 1 -- 172.21.15.107:0/1433460367 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08800cb040 msgr2=0x7f08800ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:16.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f087f7fe640 1 --2- 172.21.15.107:0/1433460367 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08800cb040 0x7f08800ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:16.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f087f7fe640 1 -- 172.21.15.107:0/1433460367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08800c84a0 con 0x7f086c005190 2023-12-04T05:42:16.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f087ffff640 1 --2- 172.21.15.107:0/1433460367 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f088015af50 0x7f08800ced60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:42:16.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f087f7fe640 1 --2- 172.21.15.107:0/1433460367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f086c005190 0x7f08800cab00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f087000e7f0 tx=0x7f087000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:16.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f087cff9640 1 -- 172.21.15.107:0/1433460367 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087000c8d0 con 0x7f086c005190 2023-12-04T05:42:16.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f087cff9640 1 -- 172.21.15.107:0/1433460367 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087000ca50 con 0x7f086c005190 2023-12-04T05:42:16.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f087cff9640 1 -- 172.21.15.107:0/1433460367 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087000ce00 con 0x7f086c005190 2023-12-04T05:42:16.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f0885e1f640 1 -- 172.21.15.107:0/1433460367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08800c8780 con 0x7f086c005190 2023-12-04T05:42:16.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f0885e1f640 1 -- 172.21.15.107:0/1433460367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08800c9260 con 0x7f086c005190 2023-12-04T05:42:16.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f0885e1f640 1 -- 172.21.15.107:0/1433460367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08800c6810 con 0x7f086c005190 2023-12-04T05:42:16.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.076+0000 7f087cff9640 1 -- 172.21.15.107:0/1433460367 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f087001d460 con 0x7f086c005190 2023-12-04T05:42:16.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.080+0000 7f087cff9640 1 --2- 172.21.15.107:0/1433460367 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0860066900 0x7f0860068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:16.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.080+0000 7f087cff9640 1 -- 172.21.15.107:0/1433460367 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f0870084fe0 con 0x7f086c005190 2023-12-04T05:42:16.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.080+0000 7f0877fff640 1 --2- 172.21.15.107:0/1433460367 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0860066900 0x7f0860068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:16.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.080+0000 7f0877fff640 1 --2- 172.21.15.107:0/1433460367 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0860066900 0x7f0860068dc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f08801f03a0 tx=0x7f0868008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:16.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.084+0000 7f087cff9640 1 -- 172.21.15.107:0/1433460367 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f08700520c0 con 0x7f086c005190 2023-12-04T05:42:16.133 INFO:tasks.workunit.client.0.smithi107.stdout:checking fcntl.h presence... yes 2023-12-04T05:42:16.134 INFO:tasks.workunit.client.0.smithi107.stdout:checking for fcntl.h... yes 2023-12-04T05:42:16.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.288+0000 7f0885e1f640 1 -- 172.21.15.107:0/1433460367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0880067870 con 0x7f086c005190 2023-12-04T05:42:16.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.288+0000 7f087cff9640 1 -- 172.21.15.107:0/1433460367 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+5771 (secure 0 0 0) 0x7f08700556e0 con 0x7f086c005190 2023-12-04T05:42:16.294 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:42:16.295 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:42:14.714915+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:42:16.295 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 18 2023-12-04T05:42:16.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.288+0000 7f0885e1f640 1 -- 172.21.15.107:0/1433460367 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0860066900 msgr2=0x7f0860068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:16.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.288+0000 7f0885e1f640 1 --2- 172.21.15.107:0/1433460367 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0860066900 0x7f0860068dc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f08801f03a0 tx=0x7f0868008040 comp rx=0 tx=0).stop 2023-12-04T05:42:16.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.288+0000 7f0885e1f640 1 -- 172.21.15.107:0/1433460367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f086c005190 msgr2=0x7f08800cab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:16.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.288+0000 7f0885e1f640 1 --2- 172.21.15.107:0/1433460367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f086c005190 0x7f08800cab00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f087000e7f0 tx=0x7f087000ecb0 comp rx=0 tx=0).stop 2023-12-04T05:42:16.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.292+0000 7f0885e1f640 1 -- 172.21.15.107:0/1433460367 shutdown_connections 2023-12-04T05:42:16.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.292+0000 7f0885e1f640 1 --2- 172.21.15.107:0/1433460367 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0860066900 0x7f0860068dc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:16.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.292+0000 7f0885e1f640 1 --2- 172.21.15.107:0/1433460367 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f088015af50 0x7f08800ced60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:16.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.292+0000 7f0885e1f640 1 --2- 172.21.15.107:0/1433460367 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08800cb040 0x7f08800ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:16.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.292+0000 7f0885e1f640 1 --2- 172.21.15.107:0/1433460367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f086c005190 0x7f08800cab00 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:16.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.292+0000 7f0885e1f640 1 -- 172.21.15.107:0/1433460367 >> 172.21.15.107:0/1433460367 conn(0x7f088006c6c0 msgr2=0x7f08801579e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:16.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.292+0000 7f0885e1f640 1 -- 172.21.15.107:0/1433460367 shutdown_connections 2023-12-04T05:42:16.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.292+0000 7f0885e1f640 1 -- 172.21.15.107:0/1433460367 wait complete. 2023-12-04T05:42:16.313 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2023-12-04T05:42:16.313 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-12-04T05:42:16.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:42:16.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.468+0000 7fcb57bb2640 1 Processor -- start 2023-12-04T05:42:16.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.468+0000 7fcb57bb2640 1 -- start start 2023-12-04T05:42:16.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.472+0000 7fcb57bb2640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcb50071c00 0x7fcb500c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:16.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.472+0000 7fcb57bb2640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcb50070380 0x7fcb50070790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:16.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.472+0000 7fcb57bb2640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb500700c0 con 0x7fcb50151180 2023-12-04T05:42:16.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.472+0000 7fcb57bb2640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb50156df0 con 0x7fcb50070380 2023-12-04T05:42:16.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.472+0000 7fcb57bb2640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb501570d0 con 0x7fcb50071c00 2023-12-04T05:42:16.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.472+0000 7fcb5614f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcb50070380 0x7fcb50070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:16.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.472+0000 7fcb5514d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcb50071c00 0x7fcb500c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:16.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.472+0000 7fcb5614f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fcb50070380 0x7fcb50070790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.107:38490/0 (socket says 172.21.15.107:38490) 2023-12-04T05:42:16.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.472+0000 7fcb5614f640 1 -- 172.21.15.107:0/1320804970 learned_addr learned my addr 172.21.15.107:0/1320804970 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:16.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.472+0000 7fcb5614f640 1 -- 172.21.15.107:0/1320804970 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcb50071c00 msgr2=0x7fcb500c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:16.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.472+0000 7fcb5614f640 1 --2- 172.21.15.107:0/1320804970 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcb50071c00 0x7fcb500c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:16.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.472+0000 7fcb5614f640 1 -- 172.21.15.107:0/1320804970 >> v1:172.21.15.195:6789/0 conn(0x7fcb50151180 legacy=0x7fcb500714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:16.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.472+0000 7fcb5614f640 1 -- 172.21.15.107:0/1320804970 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb501573b0 con 0x7fcb50070380 2023-12-04T05:42:16.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.476+0000 7fcb5614f640 1 --2- 172.21.15.107:0/1320804970 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb50070380 0x7fcb50070790 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcb48009e50 tx=0x7fcb4801c9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54af09680c6a4d86 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:16.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.476+0000 7fcb5494c640 1 -- 172.21.15.107:0/1320804970 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb480a58a0 con 0x7fcb50070380 2023-12-04T05:42:16.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.476+0000 7fcb5494c640 1 -- 172.21.15.107:0/1320804970 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb480a5a00 con 0x7fcb50070380 2023-12-04T05:42:16.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.476+0000 7fcb5494c640 1 -- 172.21.15.107:0/1320804970 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb480a5d90 con 0x7fcb50070380 2023-12-04T05:42:16.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.476+0000 7fcb57bb2640 1 -- 172.21.15.107:0/1320804970 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb50070380 msgr2=0x7fcb50070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:16.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.476+0000 7fcb57bb2640 1 --2- 172.21.15.107:0/1320804970 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb50070380 0x7fcb50070790 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcb48009e50 tx=0x7fcb4801c9e0 comp rx=0 tx=0).stop 2023-12-04T05:42:16.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.476+0000 7fcb57bb2640 1 -- 172.21.15.107:0/1320804970 shutdown_connections 2023-12-04T05:42:16.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.476+0000 7fcb57bb2640 1 --2- 172.21.15.107:0/1320804970 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb50070380 0x7fcb50070790 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:16.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.476+0000 7fcb57bb2640 1 --2- 172.21.15.107:0/1320804970 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcb50071c00 0x7fcb500c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:16.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.476+0000 7fcb57bb2640 1 -- 172.21.15.107:0/1320804970 >> 172.21.15.107:0/1320804970 conn(0x7fcb500c7ff0 msgr2=0x7fcb500c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:16.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.476+0000 7fcb57bb2640 1 -- 172.21.15.107:0/1320804970 shutdown_connections 2023-12-04T05:42:16.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.476+0000 7fcb57bb2640 1 -- 172.21.15.107:0/1320804970 wait complete. 2023-12-04T05:42:16.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.476+0000 7fcb57bb2640 1 Processor -- start 2023-12-04T05:42:16.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.476+0000 7fcb57bb2640 1 -- start start 2023-12-04T05:42:16.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb57bb2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb50070380 0x7fcb50157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:16.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb57bb2640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcb50071c00 0x7fcb50141b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:16.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb57bb2640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb50151180 0x7fcb50156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:16.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb57bb2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb501485a0 con 0x7fcb50070380 2023-12-04T05:42:16.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb57bb2640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb50142080 con 0x7fcb50151180 2023-12-04T05:42:16.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb57bb2640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb50147fe0 con 0x7fcb50071c00 2023-12-04T05:42:16.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb5594e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb50070380 0x7fcb50157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:16.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb5594e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb50070380 0x7fcb50157e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34758/0 (socket says 172.21.15.107:34758) 2023-12-04T05:42:16.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb5594e640 1 -- 172.21.15.107:0/1389068078 learned_addr learned my addr 172.21.15.107:0/1389068078 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:16.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb5514d640 1 --2- 172.21.15.107:0/1389068078 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcb50071c00 0x7fcb50141b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:16.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb5614f640 1 --2- 172.21.15.107:0/1389068078 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb50151180 0x7fcb50156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:16.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb5594e640 1 -- 172.21.15.107:0/1389068078 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcb50071c00 msgr2=0x7fcb50141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:16.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb5594e640 1 --2- 172.21.15.107:0/1389068078 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcb50071c00 0x7fcb50141b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:16.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb5594e640 1 -- 172.21.15.107:0/1389068078 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb50151180 msgr2=0x7fcb50156560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:16.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb5594e640 1 --2- 172.21.15.107:0/1389068078 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb50151180 0x7fcb50156560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:16.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb5594e640 1 -- 172.21.15.107:0/1389068078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb48009b30 con 0x7fcb50070380 2023-12-04T05:42:16.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb5594e640 1 --2- 172.21.15.107:0/1389068078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb50070380 0x7fcb50157e30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c00c3c0 tx=0x7fcb4c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:16.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb467fc640 1 -- 172.21.15.107:0/1389068078 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c0052d0 con 0x7fcb50070380 2023-12-04T05:42:16.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb467fc640 1 -- 172.21.15.107:0/1389068078 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb4c005430 con 0x7fcb50070380 2023-12-04T05:42:16.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb57bb2640 1 -- 172.21.15.107:0/1389068078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb501482c0 con 0x7fcb50070380 2023-12-04T05:42:16.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb467fc640 1 -- 172.21.15.107:0/1389068078 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c003840 con 0x7fcb50070380 2023-12-04T05:42:16.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb57bb2640 1 -- 172.21.15.107:0/1389068078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb501f33c0 con 0x7fcb50070380 2023-12-04T05:42:16.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb57bb2640 1 -- 172.21.15.107:0/1389068078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb501f2f40 con 0x7fcb50070380 2023-12-04T05:42:16.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb467fc640 1 -- 172.21.15.107:0/1389068078 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fcb4c0039a0 con 0x7fcb50070380 2023-12-04T05:42:16.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb467fc640 1 --2- 172.21.15.107:0/1389068078 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fcb340668d0 0x7fcb34068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:16.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb5514d640 1 --2- 172.21.15.107:0/1389068078 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fcb340668d0 0x7fcb34068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:16.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb467fc640 1 -- 172.21.15.107:0/1389068078 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fcb4c0839a0 con 0x7fcb50070380 2023-12-04T05:42:16.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.480+0000 7fcb5514d640 1 --2- 172.21.15.107:0/1389068078 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fcb340668d0 0x7fcb34068d90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcb5014ceb0 tx=0x7fcb40008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:16.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.484+0000 7fcb467fc640 1 -- 172.21.15.107:0/1389068078 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fcb501f2f40 con 0x7fcb50070380 2023-12-04T05:42:16.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.664+0000 7fcb57bb2640 1 -- 172.21.15.107:0/1389068078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb501f2f40 con 0x7fcb50070380 2023-12-04T05:42:16.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.664+0000 7fcb467fc640 1 -- 172.21.15.107:0/1389068078 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+5771 (secure 0 0 0) 0x7fcb501f2f40 con 0x7fcb50070380 2023-12-04T05:42:16.672 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:42:16.672 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:42:14.714915+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:42:16.672 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 18 2023-12-04T05:42:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.668+0000 7fcb57bb2640 1 -- 172.21.15.107:0/1389068078 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fcb340668d0 msgr2=0x7fcb34068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.668+0000 7fcb57bb2640 1 --2- 172.21.15.107:0/1389068078 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fcb340668d0 0x7fcb34068d90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcb5014ceb0 tx=0x7fcb40008040 comp rx=0 tx=0).stop 2023-12-04T05:42:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.668+0000 7fcb57bb2640 1 -- 172.21.15.107:0/1389068078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb50070380 msgr2=0x7fcb50157e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.668+0000 7fcb57bb2640 1 --2- 172.21.15.107:0/1389068078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb50070380 0x7fcb50157e30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c00c3c0 tx=0x7fcb4c00c880 comp rx=0 tx=0).stop 2023-12-04T05:42:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.668+0000 7fcb57bb2640 1 -- 172.21.15.107:0/1389068078 shutdown_connections 2023-12-04T05:42:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.668+0000 7fcb57bb2640 1 --2- 172.21.15.107:0/1389068078 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fcb340668d0 0x7fcb34068d90 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.668+0000 7fcb57bb2640 1 --2- 172.21.15.107:0/1389068078 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb50151180 0x7fcb50156560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.668+0000 7fcb57bb2640 1 --2- 172.21.15.107:0/1389068078 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fcb50071c00 0x7fcb50141b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.668+0000 7fcb57bb2640 1 --2- 172.21.15.107:0/1389068078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb50070380 0x7fcb50157e30 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.668+0000 7fcb57bb2640 1 -- 172.21.15.107:0/1389068078 >> 172.21.15.107:0/1389068078 conn(0x7fcb500c7ff0 msgr2=0x7fcb50150190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:16.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.668+0000 7fcb57bb2640 1 -- 172.21.15.107:0/1389068078 shutdown_connections 2023-12-04T05:42:16.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:16.668+0000 7fcb57bb2640 1 -- 172.21.15.107:0/1389068078 wait complete. 2023-12-04T05:42:16.689 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 18, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-12-04T05:39:59.032019+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 18, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4247": { "addr": "172.21.15.107:6839/993693883", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6838", "nonce": 993693883, "type": "v2" }, { "addr": "172.21.15.107:6839", "nonce": 993693883, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 18, "join_fscid": -1, "name": "a", "rank": 4, "state": "up:replay", "state_seq": 1 }, "gid_4260": { "addr": "172.21.15.107:6836/3444851515", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 3444851515, "type": "v2" }, { "addr": "172.21.15.107:6836", "nonce": 3444851515, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4260, "incarnation": 14, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 9 }, "gid_4263": { "addr": "172.21.15.107:6837/2210527429", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6835", "nonce": 2210527429, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 2210527429, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4263, "incarnation": 12, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 7 }, "gid_4279": { "addr": "172.21.15.195:6833/569093002", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6832", "nonce": 569093002, "type": "v2" }, { "addr": "172.21.15.195:6833", "nonce": 569093002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4279, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4288": { "addr": "172.21.15.195:6837/3412865477", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6835", "nonce": 3412865477, "type": "v2" }, { "addr": "172.21.15.195:6837", "nonce": 3412865477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 24, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-12-04T05:42:14.714915+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 4279, "mds_2": 4263, "mds_3": 4260, "mds_4": 4247 } } } ], "standbys": [] } 2023-12-04T05:42:17.246 INFO:tasks.workunit.client.0.smithi107.stdout:checking limits.h usability... yes 2023-12-04T05:42:17.326 INFO:tasks.workunit.client.0.smithi107.stdout:checking limits.h presence... yes 2023-12-04T05:42:17.327 INFO:tasks.workunit.client.0.smithi107.stdout:checking for limits.h... yes 2023-12-04T05:42:17.351 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stdint.h... (cached) yes 2023-12-04T05:42:17.486 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/time.h usability... yes 2023-12-04T05:42:17.562 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/time.h presence... yes 2023-12-04T05:42:17.563 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/time.h... yes 2023-12-04T05:42:17.587 INFO:tasks.workunit.client.0.smithi107.stdout:checking for unistd.h... (cached) yes 2023-12-04T05:42:17.719 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/vfs.h usability... yes 2023-12-04T05:42:17.796 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/vfs.h presence... yes 2023-12-04T05:42:17.797 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/vfs.h... yes 2023-12-04T05:42:18.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:42:18.701 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/limits.h usability... no 2023-12-04T05:42:18.790 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/limits.h presence... no 2023-12-04T05:42:18.791 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/limits.h... no 2023-12-04T05:42:18.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.816+0000 7f14797c3640 1 Processor -- start 2023-12-04T05:42:18.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.816+0000 7f14797c3640 1 -- start start 2023-12-04T05:42:18.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f14797c3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1474154020 0x7f147415a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:18.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f14797c3640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f14740d01c0 0x7f1474071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:18.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f14797c3640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f147415ad50 con 0x7f1474072220 2023-12-04T05:42:18.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f14797c3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14740ca230 con 0x7f1474154020 2023-12-04T05:42:18.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f14797c3640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14740c6770 con 0x7f14740d01c0 2023-12-04T05:42:18.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f1472ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1474154020 0x7f147415a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:18.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f1472ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1474154020 0x7f147415a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44316/0 (socket says 172.21.15.107:44316) 2023-12-04T05:42:18.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f1472ffd640 1 -- 172.21.15.107:0/1658336393 learned_addr learned my addr 172.21.15.107:0/1658336393 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:18.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f14727fc640 1 --2- 172.21.15.107:0/1658336393 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f14740d01c0 0x7f1474071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:18.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f1472ffd640 1 -- 172.21.15.107:0/1658336393 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f14740d01c0 msgr2=0x7f1474071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:18.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f1472ffd640 1 --2- 172.21.15.107:0/1658336393 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f14740d01c0 0x7f1474071ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:18.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f1472ffd640 1 -- 172.21.15.107:0/1658336393 >> v1:172.21.15.195:6789/0 conn(0x7f1474072220 legacy=0x7f14740c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:18.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f1472ffd640 1 -- 172.21.15.107:0/1658336393 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147415d710 con 0x7f1474154020 2023-12-04T05:42:18.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f1472ffd640 1 --2- 172.21.15.107:0/1658336393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1474154020 0x7f147415a810 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1464009870 tx=0x7f146401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=475d2cad4eccc0ce server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:18.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f1471ffb640 1 -- 172.21.15.107:0/1658336393 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14640a89b0 con 0x7f1474154020 2023-12-04T05:42:18.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f1471ffb640 1 -- 172.21.15.107:0/1658336393 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14640a8b10 con 0x7f1474154020 2023-12-04T05:42:18.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f1471ffb640 1 -- 172.21.15.107:0/1658336393 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14640a8ea0 con 0x7f1474154020 2023-12-04T05:42:18.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f14797c3640 1 -- 172.21.15.107:0/1658336393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1474154020 msgr2=0x7f147415a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:18.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f14797c3640 1 --2- 172.21.15.107:0/1658336393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1474154020 0x7f147415a810 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1464009870 tx=0x7f146401ecf0 comp rx=0 tx=0).stop 2023-12-04T05:42:18.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f14797c3640 1 -- 172.21.15.107:0/1658336393 shutdown_connections 2023-12-04T05:42:18.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f14797c3640 1 --2- 172.21.15.107:0/1658336393 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f14740d01c0 0x7f1474071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:18.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f14797c3640 1 --2- 172.21.15.107:0/1658336393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1474154020 0x7f147415a810 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:18.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f14797c3640 1 -- 172.21.15.107:0/1658336393 >> 172.21.15.107:0/1658336393 conn(0x7f147406c6c0 msgr2=0x7f14740cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:18.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f14797c3640 1 -- 172.21.15.107:0/1658336393 shutdown_connections 2023-12-04T05:42:18.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.820+0000 7f14797c3640 1 -- 172.21.15.107:0/1658336393 wait complete. 2023-12-04T05:42:18.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f14797c3640 1 Processor -- start 2023-12-04T05:42:18.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f14797c3640 1 -- start start 2023-12-04T05:42:18.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f14797c3640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1474072220 0x7f147414a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:18.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f14797c3640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f14740d01c0 0x7f14741f0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:18.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f14797c3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1474154020 0x7f14740ce620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:18.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f14797c3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14740c7c20 con 0x7f1474154020 2023-12-04T05:42:18.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f14797c3640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14740c8f80 con 0x7f1474072220 2023-12-04T05:42:18.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f14797c3640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f147415c9b0 con 0x7f14740d01c0 2023-12-04T05:42:18.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f14737fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1474154020 0x7f14740ce620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:18.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f14737fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1474154020 0x7f14740ce620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44320/0 (socket says 172.21.15.107:44320) 2023-12-04T05:42:18.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f14737fe640 1 -- 172.21.15.107:0/3688614685 learned_addr learned my addr 172.21.15.107:0/3688614685 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:18.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f14727fc640 1 --2- 172.21.15.107:0/3688614685 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f14740d01c0 0x7f14741f0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:18.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f1472ffd640 1 --2- 172.21.15.107:0/3688614685 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1474072220 0x7f147414a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:18.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f14737fe640 1 -- 172.21.15.107:0/3688614685 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f14740d01c0 msgr2=0x7f14741f0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:18.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f14737fe640 1 --2- 172.21.15.107:0/3688614685 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f14740d01c0 0x7f14741f0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:18.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f14737fe640 1 -- 172.21.15.107:0/3688614685 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1474072220 msgr2=0x7f147414a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:18.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f14737fe640 1 --2- 172.21.15.107:0/3688614685 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1474072220 0x7f147414a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:18.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f14737fe640 1 -- 172.21.15.107:0/3688614685 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14640094d0 con 0x7f1474154020 2023-12-04T05:42:18.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f14727fc640 1 --2- 172.21.15.107:0/3688614685 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f14740d01c0 0x7f14741f0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:42:18.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f14737fe640 1 --2- 172.21.15.107:0/3688614685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1474154020 0x7f14740ce620 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f146c00c780 tx=0x7f146c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:18.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f145b7fe640 1 -- 172.21.15.107:0/3688614685 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c004b70 con 0x7f1474154020 2023-12-04T05:42:18.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f145b7fe640 1 -- 172.21.15.107:0/3688614685 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f146c00ce90 con 0x7f1474154020 2023-12-04T05:42:18.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f14797c3640 1 -- 172.21.15.107:0/3688614685 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14741efb70 con 0x7f1474154020 2023-12-04T05:42:18.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f145b7fe640 1 -- 172.21.15.107:0/3688614685 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c00a850 con 0x7f1474154020 2023-12-04T05:42:18.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f14797c3640 1 -- 172.21.15.107:0/3688614685 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147414a930 con 0x7f1474154020 2023-12-04T05:42:18.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.824+0000 7f14797c3640 1 -- 172.21.15.107:0/3688614685 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1474067870 con 0x7f1474154020 2023-12-04T05:42:18.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.828+0000 7f145b7fe640 1 -- 172.21.15.107:0/3688614685 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f146c00a9b0 con 0x7f1474154020 2023-12-04T05:42:18.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.828+0000 7f145b7fe640 1 --2- 172.21.15.107:0/3688614685 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f14600668d0 0x7f1460068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:18.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.828+0000 7f145b7fe640 1 -- 172.21.15.107:0/3688614685 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f146c084740 con 0x7f1474154020 2023-12-04T05:42:18.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.828+0000 7f1472ffd640 1 --2- 172.21.15.107:0/3688614685 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f14600668d0 0x7f1460068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:18.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.828+0000 7f1472ffd640 1 --2- 172.21.15.107:0/3688614685 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f14600668d0 0x7f1460068d90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f14640b2000 tx=0x7f1464008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:18.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:18.832+0000 7f145b7fe640 1 -- 172.21.15.107:0/3688614685 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f146c051820 con 0x7f1474154020 2023-12-04T05:42:19.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:19.024+0000 7f14797c3640 1 -- 172.21.15.107:0/3688614685 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14740c9bc0 con 0x7f1474154020 2023-12-04T05:42:19.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:19.024+0000 7f145b7fe640 1 -- 172.21.15.107:0/3688614685 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+5771 (secure 0 0 0) 0x7f146c054e40 con 0x7f1474154020 2023-12-04T05:42:19.030 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:42:19.030 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:42:14.714915+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:42:19.030 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 18 2023-12-04T05:42:19.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:19.024+0000 7f14797c3640 1 -- 172.21.15.107:0/3688614685 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f14600668d0 msgr2=0x7f1460068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:19.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:19.024+0000 7f14797c3640 1 --2- 172.21.15.107:0/3688614685 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f14600668d0 0x7f1460068d90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f14640b2000 tx=0x7f1464008040 comp rx=0 tx=0).stop 2023-12-04T05:42:19.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:19.024+0000 7f14797c3640 1 -- 172.21.15.107:0/3688614685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1474154020 msgr2=0x7f14740ce620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:19.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:19.024+0000 7f14797c3640 1 --2- 172.21.15.107:0/3688614685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1474154020 0x7f14740ce620 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f146c00c780 tx=0x7f146c00cc40 comp rx=0 tx=0).stop 2023-12-04T05:42:19.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:19.024+0000 7f14797c3640 1 -- 172.21.15.107:0/3688614685 shutdown_connections 2023-12-04T05:42:19.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:19.024+0000 7f14797c3640 1 --2- 172.21.15.107:0/3688614685 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f14600668d0 0x7f1460068d90 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:19.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:19.024+0000 7f14797c3640 1 --2- 172.21.15.107:0/3688614685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1474154020 0x7f14740ce620 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:19.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:19.024+0000 7f14797c3640 1 --2- 172.21.15.107:0/3688614685 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f14740d01c0 0x7f14741f0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:19.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:19.024+0000 7f14797c3640 1 --2- 172.21.15.107:0/3688614685 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1474072220 0x7f147414a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:19.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:19.024+0000 7f14797c3640 1 -- 172.21.15.107:0/3688614685 >> 172.21.15.107:0/3688614685 conn(0x7f147406c6c0 msgr2=0x7f14740cba70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:19.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:19.028+0000 7f14797c3640 1 -- 172.21.15.107:0/3688614685 shutdown_connections 2023-12-04T05:42:19.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:19.028+0000 7f14797c3640 1 -- 172.21.15.107:0/3688614685 wait complete. 2023-12-04T05:42:19.049 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:42:20.445 INFO:tasks.workunit.client.0.smithi107.stdout:checking for an ANSI C-conforming const... yes 2023-12-04T05:42:20.582 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether time.h and sys/time.h may both be included... yes 2023-12-04T05:42:20.591 INFO:tasks.workunit.client.0.smithi107.stdout:checking for function prototypes... yes 2023-12-04T05:42:21.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:42:21.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.172+0000 7f7d06125640 1 Processor -- start 2023-12-04T05:42:21.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.172+0000 7f7d06125640 1 -- start start 2023-12-04T05:42:21.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.176+0000 7f7d06125640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7d00151180 0x7f7d000c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:21.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.176+0000 7f7d06125640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7d00070380 0x7f7d000707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:21.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.176+0000 7f7d06125640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d000700c0 con 0x7f7d000714f0 2023-12-04T05:42:21.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.176+0000 7f7d06125640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d00156df0 con 0x7f7d00151180 2023-12-04T05:42:21.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.176+0000 7f7d06125640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d001570d0 con 0x7f7d00070380 2023-12-04T05:42:21.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.176+0000 7f7cfeffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7d000714f0 0x7f7d00071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33428/0 (socket says 172.21.15.107:33428) 2023-12-04T05:42:21.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.176+0000 7f7cfeffd640 1 -- 172.21.15.107:0/224954386 learned_addr learned my addr 172.21.15.107:0/224954386 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:42:21.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.176+0000 7f7cff7fe640 1 --2- 172.21.15.107:0/224954386 >> v2:172.21.15.107:3300/0 conn(0x7f7d00151180 0x7f7d000c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:21.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.176+0000 7f7cfffff640 1 --2- 172.21.15.107:0/224954386 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7d00070380 0x7f7d000707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:21.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.176+0000 7f7cfe7fc640 1 -- 172.21.15.107:0/224954386 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855078072 0 0) 0x7f7d000700c0 con 0x7f7d000714f0 2023-12-04T05:42:21.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.176+0000 7f7cfe7fc640 1 -- 172.21.15.107:0/224954386 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ce8003530 con 0x7f7d000714f0 2023-12-04T05:42:21.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.176+0000 7f7cff7fe640 1 -- 172.21.15.107:0/224954386 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7d00070380 msgr2=0x7f7d000707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:21.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.176+0000 7f7cff7fe640 1 --2- 172.21.15.107:0/224954386 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7d00070380 0x7f7d000707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:21.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.176+0000 7f7cff7fe640 1 -- 172.21.15.107:0/224954386 >> v1:172.21.15.107:6789/0 conn(0x7f7d000714f0 legacy=0x7f7d00071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:21.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.176+0000 7f7cff7fe640 1 -- 172.21.15.107:0/224954386 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d001573b0 con 0x7f7d00151180 2023-12-04T05:42:21.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.176+0000 7f7cfffff640 1 --2- 172.21.15.107:0/224954386 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7d00070380 0x7f7d000707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:42:21.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7cff7fe640 1 --2- 172.21.15.107:0/224954386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d00151180 0x7f7d000c79e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7cf00097c0 tx=0x7f7cf001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd247752559c848d server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:21.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7cfe7fc640 1 -- 172.21.15.107:0/224954386 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf00a8970 con 0x7f7d00151180 2023-12-04T05:42:21.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7cfe7fc640 1 -- 172.21.15.107:0/224954386 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf00a8ad0 con 0x7f7d00151180 2023-12-04T05:42:21.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7cfe7fc640 1 -- 172.21.15.107:0/224954386 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf00a8e40 con 0x7f7d00151180 2023-12-04T05:42:21.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7d06125640 1 -- 172.21.15.107:0/224954386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d00151180 msgr2=0x7f7d000c79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:21.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7d06125640 1 --2- 172.21.15.107:0/224954386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d00151180 0x7f7d000c79e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7cf00097c0 tx=0x7f7cf001ec70 comp rx=0 tx=0).stop 2023-12-04T05:42:21.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7d06125640 1 -- 172.21.15.107:0/224954386 shutdown_connections 2023-12-04T05:42:21.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7d06125640 1 --2- 172.21.15.107:0/224954386 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7d00070380 0x7f7d000707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:21.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7d06125640 1 --2- 172.21.15.107:0/224954386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d00151180 0x7f7d000c79e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:21.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7d06125640 1 -- 172.21.15.107:0/224954386 >> 172.21.15.107:0/224954386 conn(0x7f7d000c7ff0 msgr2=0x7f7d000c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:21.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7d06125640 1 -- 172.21.15.107:0/224954386 shutdown_connections 2023-12-04T05:42:21.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7d06125640 1 -- 172.21.15.107:0/224954386 wait complete. 2023-12-04T05:42:21.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7d06125640 1 Processor -- start 2023-12-04T05:42:21.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7d06125640 1 -- start start 2023-12-04T05:42:21.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7d06125640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7d00070380 0x7f7d001f0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:21.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7d06125640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7d000714f0 0x7f7d001464e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:21.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7d06125640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d00151180 0x7f7d00146a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:21.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7d06125640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d001515c0 con 0x7f7d00151180 2023-12-04T05:42:21.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7d06125640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d001f14d0 con 0x7f7d00070380 2023-12-04T05:42:21.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7d06125640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d00152b20 con 0x7f7d000714f0 2023-12-04T05:42:21.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7cfffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d00151180 0x7f7d00146a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:21.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7cfffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d00151180 0x7f7d00146a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44342/0 (socket says 172.21.15.107:44342) 2023-12-04T05:42:21.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7cfffff640 1 -- 172.21.15.107:0/3820553668 learned_addr learned my addr 172.21.15.107:0/3820553668 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:21.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7cff7fe640 1 --2- 172.21.15.107:0/3820553668 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7d00070380 0x7f7d001f0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7cfffff640 1 -- 172.21.15.107:0/3820553668 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7d000714f0 msgr2=0x7f7d001464e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7cfffff640 1 --2- 172.21.15.107:0/3820553668 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7d000714f0 0x7f7d001464e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7cfffff640 1 -- 172.21.15.107:0/3820553668 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7d00070380 msgr2=0x7f7d001f0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7cfffff640 1 --2- 172.21.15.107:0/3820553668 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7d00070380 0x7f7d001f0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7cfffff640 1 -- 172.21.15.107:0/3820553668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf0009420 con 0x7f7d00151180 2023-12-04T05:42:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7cff7fe640 1 --2- 172.21.15.107:0/3820553668 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7d00070380 0x7f7d001f0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:42:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7cfffff640 1 --2- 172.21.15.107:0/3820553668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d00151180 0x7f7d00146a20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8007ef0 tx=0x7f7cf800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7ce7fff640 1 -- 172.21.15.107:0/3820553668 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8003ba0 con 0x7f7d00151180 2023-12-04T05:42:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7ce7fff640 1 -- 172.21.15.107:0/3820553668 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf800f040 con 0x7f7d00151180 2023-12-04T05:42:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7d06125640 1 -- 172.21.15.107:0/3820553668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d00152e00 con 0x7f7d00151180 2023-12-04T05:42:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7ce7fff640 1 -- 172.21.15.107:0/3820553668 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf800de80 con 0x7f7d00151180 2023-12-04T05:42:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.180+0000 7f7d06125640 1 -- 172.21.15.107:0/3820553668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d0014f0a0 con 0x7f7d00151180 2023-12-04T05:42:21.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.184+0000 7f7d06125640 1 -- 172.21.15.107:0/3820553668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d0014f2c0 con 0x7f7d00151180 2023-12-04T05:42:21.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.184+0000 7f7ce7fff640 1 -- 172.21.15.107:0/3820553668 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f7cf8004030 con 0x7f7d00151180 2023-12-04T05:42:21.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.184+0000 7f7ce7fff640 1 --2- 172.21.15.107:0/3820553668 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7cec066900 0x7f7cec068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:21.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.184+0000 7f7ce7fff640 1 -- 172.21.15.107:0/3820553668 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f7cf80847e0 con 0x7f7d00151180 2023-12-04T05:42:21.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.184+0000 7f7cff7fe640 1 --2- 172.21.15.107:0/3820553668 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7cec066900 0x7f7cec068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:21.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.184+0000 7f7cff7fe640 1 --2- 172.21.15.107:0/3820553668 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7cec066900 0x7f7cec068dc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7cf00b2000 tx=0x7f7cf00aa070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:21.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.188+0000 7f7ce7fff640 1 -- 172.21.15.107:0/3820553668 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7cf8051960 con 0x7f7d00151180 2023-12-04T05:42:21.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.376+0000 7f7d06125640 1 -- 172.21.15.107:0/3820553668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d00067870 con 0x7f7d00151180 2023-12-04T05:42:21.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.376+0000 7f7ce7fff640 1 -- 172.21.15.107:0/3820553668 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+5773 (secure 0 0 0) 0x7f7cf8054f80 con 0x7f7d00151180 2023-12-04T05:42:21.382 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:42:21.382 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:42:19.735999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:resolve","state_seq":38,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:42:21.382 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 19 2023-12-04T05:42:21.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.376+0000 7f7d06125640 1 -- 172.21.15.107:0/3820553668 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7cec066900 msgr2=0x7f7cec068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:21.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.376+0000 7f7d06125640 1 --2- 172.21.15.107:0/3820553668 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7cec066900 0x7f7cec068dc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7cf00b2000 tx=0x7f7cf00aa070 comp rx=0 tx=0).stop 2023-12-04T05:42:21.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.376+0000 7f7d06125640 1 -- 172.21.15.107:0/3820553668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d00151180 msgr2=0x7f7d00146a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:21.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.376+0000 7f7d06125640 1 --2- 172.21.15.107:0/3820553668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d00151180 0x7f7d00146a20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8007ef0 tx=0x7f7cf800d4a0 comp rx=0 tx=0).stop 2023-12-04T05:42:21.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.376+0000 7f7d06125640 1 -- 172.21.15.107:0/3820553668 shutdown_connections 2023-12-04T05:42:21.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.376+0000 7f7d06125640 1 --2- 172.21.15.107:0/3820553668 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f7cec066900 0x7f7cec068dc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:21.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.376+0000 7f7d06125640 1 --2- 172.21.15.107:0/3820553668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d00151180 0x7f7d00146a20 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:21.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.376+0000 7f7d06125640 1 --2- 172.21.15.107:0/3820553668 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7d000714f0 0x7f7d001464e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:21.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.376+0000 7f7d06125640 1 --2- 172.21.15.107:0/3820553668 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7d00070380 0x7f7d001f0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:21.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.376+0000 7f7d06125640 1 -- 172.21.15.107:0/3820553668 >> 172.21.15.107:0/3820553668 conn(0x7f7d000c7ff0 msgr2=0x7f7d000707d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:21.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.376+0000 7f7d06125640 1 -- 172.21.15.107:0/3820553668 shutdown_connections 2023-12-04T05:42:21.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:21.376+0000 7f7d06125640 1 -- 172.21.15.107:0/3820553668 wait complete. 2023-12-04T05:42:21.402 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:42:21.442 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether setvbuf arguments are reversed... no 2023-12-04T05:42:21.622 INFO:tasks.workunit.client.0.smithi107.stdout:checking for system... yes 2023-12-04T05:42:21.964 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gettimeofday... yes 2023-12-04T05:42:22.291 INFO:tasks.workunit.client.0.smithi107.stdout:checking for mkdir... yes 2023-12-04T05:42:22.488 INFO:tasks.workunit.client.0.smithi107.stdout:checking for strerror... yes 2023-12-04T05:42:23.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:42:23.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.532+0000 7fb25e2de640 1 Processor -- start 2023-12-04T05:42:23.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.532+0000 7fb25e2de640 1 -- start start 2023-12-04T05:42:23.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.532+0000 7fb25e2de640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb25815e6b0 0x7fb25815ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:23.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.532+0000 7fb25e2de640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb2580c8c10 0x7fb25815efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:23.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.532+0000 7fb25e2de640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2580c5ef0 con 0x7fb2581581d0 2023-12-04T05:42:23.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.532+0000 7fb25e2de640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2580c6290 con 0x7fb25815e6b0 2023-12-04T05:42:23.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.532+0000 7fb25e2de640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2580c25e0 con 0x7fb2580c8c10 2023-12-04T05:42:23.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.532+0000 7fb257fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb25815e6b0 0x7fb25815ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:23.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.532+0000 7fb25c87b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb2581581d0 0x7fb258158620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33444/0 (socket says 172.21.15.107:33444) 2023-12-04T05:42:23.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.532+0000 7fb25c87b640 1 -- 172.21.15.107:0/1366470567 learned_addr learned my addr 172.21.15.107:0/1366470567 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:42:23.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.532+0000 7fb2577fe640 1 --2- 172.21.15.107:0/1366470567 >> v2:172.21.15.195:3300/0 conn(0x7fb2580c8c10 0x7fb25815efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:23.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.532+0000 7fb257fff640 1 -- 172.21.15.107:0/1366470567 >> v2:172.21.15.195:3300/0 conn(0x7fb2580c8c10 msgr2=0x7fb25815efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:23.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.532+0000 7fb257fff640 1 --2- 172.21.15.107:0/1366470567 >> v2:172.21.15.195:3300/0 conn(0x7fb2580c8c10 0x7fb25815efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:23.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.532+0000 7fb257fff640 1 -- 172.21.15.107:0/1366470567 >> v1:172.21.15.107:6789/0 conn(0x7fb2581581d0 legacy=0x7fb258158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:23.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.532+0000 7fb257fff640 1 -- 172.21.15.107:0/1366470567 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25806dfa0 con 0x7fb25815e6b0 2023-12-04T05:42:23.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.532+0000 7fb256ffd640 1 -- 172.21.15.107:0/1366470567 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621635877 0 0) 0x7fb2580c5ef0 con 0x7fb2581581d0 2023-12-04T05:42:23.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.536+0000 7fb257fff640 1 --2- 172.21.15.107:0/1366470567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb25815e6b0 0x7fb25815ea90 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb258074700 tx=0x7fb24001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c531a4e2e7bcebbb server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:23.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.536+0000 7fb256ffd640 1 -- 172.21.15.107:0/1366470567 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2400a8950 con 0x7fb25815e6b0 2023-12-04T05:42:23.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.536+0000 7fb256ffd640 1 -- 172.21.15.107:0/1366470567 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2400a8ab0 con 0x7fb25815e6b0 2023-12-04T05:42:23.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.536+0000 7fb256ffd640 1 -- 172.21.15.107:0/1366470567 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2400a8e40 con 0x7fb25815e6b0 2023-12-04T05:42:23.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.536+0000 7fb25e2de640 1 -- 172.21.15.107:0/1366470567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb25815e6b0 msgr2=0x7fb25815ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:23.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.536+0000 7fb25e2de640 1 --2- 172.21.15.107:0/1366470567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb25815e6b0 0x7fb25815ea90 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb258074700 tx=0x7fb24001ec50 comp rx=0 tx=0).stop 2023-12-04T05:42:23.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.536+0000 7fb25e2de640 1 -- 172.21.15.107:0/1366470567 shutdown_connections 2023-12-04T05:42:23.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.536+0000 7fb25e2de640 1 --2- 172.21.15.107:0/1366470567 >> v2:172.21.15.195:3300/0 conn(0x7fb2580c8c10 0x7fb25815efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:23.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.536+0000 7fb25e2de640 1 --2- 172.21.15.107:0/1366470567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb25815e6b0 0x7fb25815ea90 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:23.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.536+0000 7fb25e2de640 1 -- 172.21.15.107:0/1366470567 >> 172.21.15.107:0/1366470567 conn(0x7fb2580c7ff0 msgr2=0x7fb2580c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:23.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.536+0000 7fb25e2de640 1 -- 172.21.15.107:0/1366470567 shutdown_connections 2023-12-04T05:42:23.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.536+0000 7fb25e2de640 1 -- 172.21.15.107:0/1366470567 wait complete. 2023-12-04T05:42:23.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.536+0000 7fb25e2de640 1 Processor -- start 2023-12-04T05:42:23.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.536+0000 7fb25e2de640 1 -- start start 2023-12-04T05:42:23.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb25e2de640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb2580c8c10 0x7fb2581f7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:23.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb25e2de640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2581581d0 0x7fb258070f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:23.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb25e2de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb25815e6b0 0x7fb25814e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:23.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb25e2de640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2581f7c70 con 0x7fb25815e6b0 2023-12-04T05:42:23.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb25e2de640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb258151640 con 0x7fb2581581d0 2023-12-04T05:42:23.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb25e2de640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb258151080 con 0x7fb2580c8c10 2023-12-04T05:42:23.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb25c87b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb25815e6b0 0x7fb25814e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:23.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb25c87b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb25815e6b0 0x7fb25814e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44364/0 (socket says 172.21.15.107:44364) 2023-12-04T05:42:23.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb25c87b640 1 -- 172.21.15.107:0/3300369042 learned_addr learned my addr 172.21.15.107:0/3300369042 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:23.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb2577fe640 1 --2- 172.21.15.107:0/3300369042 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2581581d0 0x7fb258070f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:23.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb257fff640 1 --2- 172.21.15.107:0/3300369042 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb2580c8c10 0x7fb2581f7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:23.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb25c87b640 1 -- 172.21.15.107:0/3300369042 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb2580c8c10 msgr2=0x7fb2581f7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:23.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb25c87b640 1 --2- 172.21.15.107:0/3300369042 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb2580c8c10 0x7fb2581f7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:23.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb25c87b640 1 -- 172.21.15.107:0/3300369042 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2581581d0 msgr2=0x7fb258070f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:23.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb25c87b640 1 --2- 172.21.15.107:0/3300369042 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2581581d0 0x7fb258070f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:23.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb25c87b640 1 -- 172.21.15.107:0/3300369042 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb240009420 con 0x7fb25815e6b0 2023-12-04T05:42:23.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb25c87b640 1 --2- 172.21.15.107:0/3300369042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb25815e6b0 0x7fb25814e810 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb250001570 tx=0x7fb25000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:23.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb254ff9640 1 -- 172.21.15.107:0/3300369042 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25000a640 con 0x7fb25815e6b0 2023-12-04T05:42:23.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb254ff9640 1 -- 172.21.15.107:0/3300369042 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25000a7c0 con 0x7fb25815e6b0 2023-12-04T05:42:23.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb25e2de640 1 -- 172.21.15.107:0/3300369042 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb258151360 con 0x7fb25815e6b0 2023-12-04T05:42:23.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb254ff9640 1 -- 172.21.15.107:0/3300369042 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25000ab70 con 0x7fb25815e6b0 2023-12-04T05:42:23.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb25e2de640 1 -- 172.21.15.107:0/3300369042 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb258162500 con 0x7fb25815e6b0 2023-12-04T05:42:23.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb25e2de640 1 -- 172.21.15.107:0/3300369042 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb258162080 con 0x7fb25815e6b0 2023-12-04T05:42:23.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.540+0000 7fb254ff9640 1 -- 172.21.15.107:0/3300369042 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fb25000acd0 con 0x7fb25815e6b0 2023-12-04T05:42:23.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.544+0000 7fb254ff9640 1 --2- 172.21.15.107:0/3300369042 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb2380668d0 0x7fb238068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:23.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.544+0000 7fb254ff9640 1 -- 172.21.15.107:0/3300369042 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fb250083c00 con 0x7fb25815e6b0 2023-12-04T05:42:23.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.544+0000 7fb257fff640 1 --2- 172.21.15.107:0/3300369042 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb2380668d0 0x7fb238068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:23.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.544+0000 7fb257fff640 1 --2- 172.21.15.107:0/3300369042 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb2380668d0 0x7fb238068d90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb2400b2000 tx=0x7fb240008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:23.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.544+0000 7fb254ff9640 1 -- 172.21.15.107:0/3300369042 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb250050f00 con 0x7fb25815e6b0 2023-12-04T05:42:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.740+0000 7fb25e2de640 1 -- 172.21.15.107:0/3300369042 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb258162080 con 0x7fb25815e6b0 2023-12-04T05:42:23.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.740+0000 7fb254ff9640 1 -- 172.21.15.107:0/3300369042 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+5773 (secure 0 0 0) 0x7fb258162080 con 0x7fb25815e6b0 2023-12-04T05:42:23.746 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:42:23.746 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:42:19.735999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:resolve","state_seq":38,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:42:23.746 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 19 2023-12-04T05:42:23.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.740+0000 7fb25e2de640 1 -- 172.21.15.107:0/3300369042 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb2380668d0 msgr2=0x7fb238068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:23.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.740+0000 7fb25e2de640 1 --2- 172.21.15.107:0/3300369042 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb2380668d0 0x7fb238068d90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb2400b2000 tx=0x7fb240008040 comp rx=0 tx=0).stop 2023-12-04T05:42:23.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.740+0000 7fb25e2de640 1 -- 172.21.15.107:0/3300369042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb25815e6b0 msgr2=0x7fb25814e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:23.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.740+0000 7fb25e2de640 1 --2- 172.21.15.107:0/3300369042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb25815e6b0 0x7fb25814e810 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb250001570 tx=0x7fb25000cca0 comp rx=0 tx=0).stop 2023-12-04T05:42:23.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.740+0000 7fb25e2de640 1 -- 172.21.15.107:0/3300369042 shutdown_connections 2023-12-04T05:42:23.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.740+0000 7fb25e2de640 1 --2- 172.21.15.107:0/3300369042 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb2380668d0 0x7fb238068d90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:23.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.740+0000 7fb25e2de640 1 --2- 172.21.15.107:0/3300369042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb25815e6b0 0x7fb25814e810 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:23.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.740+0000 7fb25e2de640 1 --2- 172.21.15.107:0/3300369042 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb2581581d0 0x7fb258070f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:23.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.740+0000 7fb25e2de640 1 --2- 172.21.15.107:0/3300369042 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb2580c8c10 0x7fb2581f7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:23.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.740+0000 7fb25e2de640 1 -- 172.21.15.107:0/3300369042 >> 172.21.15.107:0/3300369042 conn(0x7fb2580c7ff0 msgr2=0x7fb25815d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:23.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.740+0000 7fb25e2de640 1 -- 172.21.15.107:0/3300369042 shutdown_connections 2023-12-04T05:42:23.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:23.740+0000 7fb25e2de640 1 -- 172.21.15.107:0/3300369042 wait complete. 2023-12-04T05:42:23.764 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:42:24.219 INFO:tasks.workunit.client.0.smithi107.stdout:checking for open64... yes 2023-12-04T05:42:24.885 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stat64... yes 2023-12-04T05:42:25.092 INFO:tasks.workunit.client.0.smithi107.stdout:checking for fseeko64... yes 2023-12-04T05:42:25.285 INFO:tasks.workunit.client.0.smithi107.stdout:checking for lrand48_r... yes 2023-12-04T05:42:25.483 INFO:tasks.workunit.client.0.smithi107.stdout:checking for srand48_r... yes 2023-12-04T05:42:25.540 INFO:tasks.workunit.client.0.smithi107.stdout:configure: creating ./config.status 2023-12-04T05:42:25.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:42:25.837 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: creating Makefile 2023-12-04T05:42:25.879 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: creating config.h 2023-12-04T05:42:25.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.884+0000 7f3ee2485640 1 Processor -- start 2023-12-04T05:42:25.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.884+0000 7f3ee2485640 1 -- start start 2023-12-04T05:42:25.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.884+0000 7f3ee2485640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3edc0c3990 0x7f3edc06e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:25.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.884+0000 7f3ee2485640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3edc06a5b0 0x7f3edc15be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:25.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.884+0000 7f3ee2485640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3edc15b9e0 con 0x7f3edc0c3d70 2023-12-04T05:42:25.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.884+0000 7f3ee2485640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3edc0c4dc0 con 0x7f3edc06a5b0 2023-12-04T05:42:25.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.884+0000 7f3ee2485640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3edc06b100 con 0x7f3edc0c3990 2023-12-04T05:42:25.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.888+0000 7f3edb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3edc06a5b0 0x7f3edc15be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:25.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.888+0000 7f3edb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3edc06a5b0 0x7f3edc15be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44366/0 (socket says 172.21.15.107:44366) 2023-12-04T05:42:25.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.888+0000 7f3edb7fe640 1 -- 172.21.15.107:0/218521475 learned_addr learned my addr 172.21.15.107:0/218521475 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:25.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.888+0000 7f3edbfff640 1 --2- 172.21.15.107:0/218521475 >> v2:172.21.15.195:3300/0 conn(0x7f3edc0c3990 0x7f3edc06e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:25.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.888+0000 7f3edb7fe640 1 -- 172.21.15.107:0/218521475 >> v2:172.21.15.195:3300/0 conn(0x7f3edc0c3990 msgr2=0x7f3edc06e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:25.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.888+0000 7f3edb7fe640 1 --2- 172.21.15.107:0/218521475 >> v2:172.21.15.195:3300/0 conn(0x7f3edc0c3990 0x7f3edc06e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:25.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.888+0000 7f3edb7fe640 1 -- 172.21.15.107:0/218521475 >> v1:172.21.15.195:6789/0 conn(0x7f3edc0c3d70 legacy=0x7f3edc158780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:25.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.888+0000 7f3edb7fe640 1 -- 172.21.15.107:0/218521475 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3edc06ab40 con 0x7f3edc06a5b0 2023-12-04T05:42:25.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.888+0000 7f3edb7fe640 1 --2- 172.21.15.107:0/218521475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3edc06a5b0 0x7f3edc15be70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3edc0ca030 tx=0x7f3ed001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb41df76d6685143 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:25.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.888+0000 7f3edaffd640 1 -- 172.21.15.107:0/218521475 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed00ab600 con 0x7f3edc06a5b0 2023-12-04T05:42:25.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.888+0000 7f3edaffd640 1 -- 172.21.15.107:0/218521475 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed00a7050 con 0x7f3edc06a5b0 2023-12-04T05:42:25.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.888+0000 7f3edaffd640 1 -- 172.21.15.107:0/218521475 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed00ab980 con 0x7f3edc06a5b0 2023-12-04T05:42:25.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.888+0000 7f3ee2485640 1 -- 172.21.15.107:0/218521475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3edc06a5b0 msgr2=0x7f3edc15be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:25.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.888+0000 7f3ee2485640 1 --2- 172.21.15.107:0/218521475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3edc06a5b0 0x7f3edc15be70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3edc0ca030 tx=0x7f3ed001c430 comp rx=0 tx=0).stop 2023-12-04T05:42:25.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.888+0000 7f3ee2485640 1 -- 172.21.15.107:0/218521475 shutdown_connections 2023-12-04T05:42:25.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.888+0000 7f3ee2485640 1 --2- 172.21.15.107:0/218521475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3edc06a5b0 0x7f3edc15be70 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:25.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.888+0000 7f3ee2485640 1 --2- 172.21.15.107:0/218521475 >> v2:172.21.15.195:3300/0 conn(0x7f3edc0c3990 0x7f3edc06e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:25.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.888+0000 7f3ee2485640 1 -- 172.21.15.107:0/218521475 >> 172.21.15.107:0/218521475 conn(0x7f3edc06b6c0 msgr2=0x7f3edc06bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:25.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.888+0000 7f3ee2485640 1 -- 172.21.15.107:0/218521475 shutdown_connections 2023-12-04T05:42:25.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.888+0000 7f3ee2485640 1 -- 172.21.15.107:0/218521475 wait complete. 2023-12-04T05:42:25.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3ee2485640 1 Processor -- start 2023-12-04T05:42:25.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3ee2485640 1 -- start start 2023-12-04T05:42:25.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3ee2485640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3edc06a5b0 0x7f3edc0c2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:25.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3ee2485640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3edc0c3990 0x7f3edc0c0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:25.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3ee2485640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3edc0c3d70 0x7f3edc0c0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:25.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3ee2485640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc1449b0 con 0x7f3edc06a5b0 2023-12-04T05:42:25.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3ee2485640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc0c1c50 con 0x7f3edc0c3990 2023-12-04T05:42:25.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3ee2485640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc1f3830 con 0x7f3edc0c3d70 2023-12-04T05:42:25.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3edbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3edc06a5b0 0x7f3edc0c2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:25.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3edbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3edc06a5b0 0x7f3edc0c2d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44382/0 (socket says 172.21.15.107:44382) 2023-12-04T05:42:25.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3edbfff640 1 -- 172.21.15.107:0/4225937901 learned_addr learned my addr 172.21.15.107:0/4225937901 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:25.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3edb7fe640 1 --2- 172.21.15.107:0/4225937901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3edc0c3990 0x7f3edc0c0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:25.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3ee0a22640 1 --2- 172.21.15.107:0/4225937901 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3edc0c3d70 0x7f3edc0c0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:25.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3edbfff640 1 -- 172.21.15.107:0/4225937901 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3edc0c3d70 msgr2=0x7f3edc0c0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:25.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3edbfff640 1 --2- 172.21.15.107:0/4225937901 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3edc0c3d70 0x7f3edc0c0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:25.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3edbfff640 1 -- 172.21.15.107:0/4225937901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3edc0c3990 msgr2=0x7f3edc0c0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:25.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3edbfff640 1 --2- 172.21.15.107:0/4225937901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3edc0c3990 0x7f3edc0c0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:25.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3edbfff640 1 -- 172.21.15.107:0/4225937901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed00094d0 con 0x7f3edc06a5b0 2023-12-04T05:42:25.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3edbfff640 1 --2- 172.21.15.107:0/4225937901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3edc06a5b0 0x7f3edc0c2d50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3ed400d800 tx=0x7f3ed400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:25.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3ed8ff9640 1 -- 172.21.15.107:0/4225937901 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed400b6c0 con 0x7f3edc06a5b0 2023-12-04T05:42:25.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3ed8ff9640 1 -- 172.21.15.107:0/4225937901 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed400b820 con 0x7f3edc06a5b0 2023-12-04T05:42:25.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3ee2485640 1 -- 172.21.15.107:0/4225937901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3edc1f3d60 con 0x7f3edc06a5b0 2023-12-04T05:42:25.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3ed8ff9640 1 -- 172.21.15.107:0/4225937901 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed400bb90 con 0x7f3edc06a5b0 2023-12-04T05:42:25.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3ee2485640 1 -- 172.21.15.107:0/4225937901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3edc159080 con 0x7f3edc06a5b0 2023-12-04T05:42:25.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3ee2485640 1 -- 172.21.15.107:0/4225937901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3edc067ec0 con 0x7f3edc06a5b0 2023-12-04T05:42:25.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.892+0000 7f3ed8ff9640 1 -- 172.21.15.107:0/4225937901 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f3ed400bcf0 con 0x7f3edc06a5b0 2023-12-04T05:42:25.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.896+0000 7f3ed8ff9640 1 --2- 172.21.15.107:0/4225937901 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3ec80668d0 0x7f3ec8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:25.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.896+0000 7f3ed8ff9640 1 -- 172.21.15.107:0/4225937901 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f3ed4084a10 con 0x7f3edc06a5b0 2023-12-04T05:42:25.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.896+0000 7f3edb7fe640 1 --2- 172.21.15.107:0/4225937901 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3ec80668d0 0x7f3ec8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:25.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.896+0000 7f3edb7fe640 1 --2- 172.21.15.107:0/4225937901 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3ec80668d0 0x7f3ec8068d90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3ed00ad000 tx=0x7f3ed0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:25.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:25.900+0000 7f3ed8ff9640 1 -- 172.21.15.107:0/4225937901 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3ed4051c10 con 0x7f3edc06a5b0 2023-12-04T05:42:26.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:26.092+0000 7f3ee2485640 1 -- 172.21.15.107:0/4225937901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3edc06c1e0 con 0x7f3edc06a5b0 2023-12-04T05:42:26.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:26.092+0000 7f3ed8ff9640 1 -- 172.21.15.107:0/4225937901 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+5775 (secure 0 0 0) 0x7f3ed4055230 con 0x7f3edc06a5b0 2023-12-04T05:42:26.099 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:42:26.099 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:42:25.729333+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:reconnect","state_seq":40,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:42:26.099 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 20 2023-12-04T05:42:26.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:26.096+0000 7f3ee2485640 1 -- 172.21.15.107:0/4225937901 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3ec80668d0 msgr2=0x7f3ec8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:26.096+0000 7f3ee2485640 1 --2- 172.21.15.107:0/4225937901 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3ec80668d0 0x7f3ec8068d90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3ed00ad000 tx=0x7f3ed0009040 comp rx=0 tx=0).stop 2023-12-04T05:42:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:26.096+0000 7f3ee2485640 1 -- 172.21.15.107:0/4225937901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3edc06a5b0 msgr2=0x7f3edc0c2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:26.096+0000 7f3ee2485640 1 --2- 172.21.15.107:0/4225937901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3edc06a5b0 0x7f3edc0c2d50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3ed400d800 tx=0x7f3ed400dcc0 comp rx=0 tx=0).stop 2023-12-04T05:42:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:26.096+0000 7f3ee2485640 1 -- 172.21.15.107:0/4225937901 shutdown_connections 2023-12-04T05:42:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:26.096+0000 7f3ee2485640 1 --2- 172.21.15.107:0/4225937901 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3ec80668d0 0x7f3ec8068d90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:26.096+0000 7f3ee2485640 1 --2- 172.21.15.107:0/4225937901 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3edc0c3d70 0x7f3edc0c0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:26.096+0000 7f3ee2485640 1 --2- 172.21.15.107:0/4225937901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3edc0c3990 0x7f3edc0c0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:26.096+0000 7f3ee2485640 1 --2- 172.21.15.107:0/4225937901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3edc06a5b0 0x7f3edc0c2d50 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:26.096+0000 7f3ee2485640 1 -- 172.21.15.107:0/4225937901 >> 172.21.15.107:0/4225937901 conn(0x7f3edc06b6c0 msgr2=0x7f3edc06ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:26.096+0000 7f3ee2485640 1 -- 172.21.15.107:0/4225937901 shutdown_connections 2023-12-04T05:42:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:26.096+0000 7f3ee2485640 1 -- 172.21.15.107:0/4225937901 wait complete. 2023-12-04T05:42:26.117 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:42:26.414 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: executing depfiles commands 2023-12-04T05:42:27.599 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: executing default commands 2023-12-04T05:42:27.724 INFO:tasks.workunit.client.0.smithi107.stderr:+ make 2023-12-04T05:42:27.800 INFO:tasks.workunit.client.0.smithi107.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2023-12-04T05:42:27.809 INFO:tasks.workunit.client.0.smithi107.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2023-12-04T05:42:27.811 INFO:tasks.workunit.client.0.smithi107.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2023-12-04T05:42:27.811 INFO:tasks.workunit.client.0.smithi107.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2023-12-04T05:42:27.811 INFO:tasks.workunit.client.0.smithi107.stderr: to install the `Automake' and `Perl' packages. Grab them from 2023-12-04T05:42:27.811 INFO:tasks.workunit.client.0.smithi107.stderr: any GNU archive site. 2023-12-04T05:42:27.818 INFO:tasks.workunit.client.0.smithi107.stdout: cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2023-12-04T05:42:27.825 INFO:tasks.workunit.client.0.smithi107.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2023-12-04T05:42:27.826 INFO:tasks.workunit.client.0.smithi107.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2023-12-04T05:42:27.826 INFO:tasks.workunit.client.0.smithi107.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2023-12-04T05:42:27.826 INFO:tasks.workunit.client.0.smithi107.stderr: You might want to install the `Automake' and `Perl' packages. 2023-12-04T05:42:27.826 INFO:tasks.workunit.client.0.smithi107.stderr: Grab them from any GNU archive site. 2023-12-04T05:42:28.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:42:28.208 INFO:tasks.workunit.client.0.smithi107.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2023-12-04T05:42:28.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.240+0000 7f551507b640 1 Processor -- start 2023-12-04T05:42:28.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.240+0000 7f551507b640 1 -- start start 2023-12-04T05:42:28.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.240+0000 7f551507b640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f55100c3990 0x7f551006e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:28.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.240+0000 7f551507b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f551006a5b0 0x7f551015be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:28.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.240+0000 7f551507b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f551015b9e0 con 0x7f55100c3d70 2023-12-04T05:42:28.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.240+0000 7f551507b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55100c4dc0 con 0x7f551006a5b0 2023-12-04T05:42:28.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.240+0000 7f551507b640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f551006b100 con 0x7f55100c3990 2023-12-04T05:42:28.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.240+0000 7f550e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f551006a5b0 0x7f551015be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:28.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.240+0000 7f550e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f551006a5b0 0x7f551015be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47404/0 (socket says 172.21.15.107:47404) 2023-12-04T05:42:28.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.240+0000 7f550e59c640 1 -- 172.21.15.107:0/2325550182 learned_addr learned my addr 172.21.15.107:0/2325550182 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:28.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.240+0000 7f550ed9d640 1 --2- 172.21.15.107:0/2325550182 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f55100c3990 0x7f551006e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:28.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.240+0000 7f550e59c640 1 -- 172.21.15.107:0/2325550182 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f55100c3990 msgr2=0x7f551006e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:28.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.240+0000 7f550e59c640 1 --2- 172.21.15.107:0/2325550182 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f55100c3990 0x7f551006e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:28.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.240+0000 7f550e59c640 1 -- 172.21.15.107:0/2325550182 >> v1:172.21.15.195:6789/0 conn(0x7f55100c3d70 legacy=0x7f5510158780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:28.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.240+0000 7f550e59c640 1 -- 172.21.15.107:0/2325550182 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f551006ab40 con 0x7f551006a5b0 2023-12-04T05:42:28.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.244+0000 7f550e59c640 1 --2- 172.21.15.107:0/2325550182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f551006a5b0 0x7f551015be70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f54f8009870 tx=0x7f54f801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e630b66d777512f server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:28.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.244+0000 7f550dd9b640 1 -- 172.21.15.107:0/2325550182 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f80ab600 con 0x7f551006a5b0 2023-12-04T05:42:28.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.244+0000 7f550dd9b640 1 -- 172.21.15.107:0/2325550182 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f80a7050 con 0x7f551006a5b0 2023-12-04T05:42:28.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.244+0000 7f550dd9b640 1 -- 172.21.15.107:0/2325550182 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f80ab980 con 0x7f551006a5b0 2023-12-04T05:42:28.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.244+0000 7f551507b640 1 -- 172.21.15.107:0/2325550182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f551006a5b0 msgr2=0x7f551015be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:28.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.244+0000 7f551507b640 1 --2- 172.21.15.107:0/2325550182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f551006a5b0 0x7f551015be70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f54f8009870 tx=0x7f54f801c470 comp rx=0 tx=0).stop 2023-12-04T05:42:28.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.244+0000 7f551507b640 1 -- 172.21.15.107:0/2325550182 shutdown_connections 2023-12-04T05:42:28.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.244+0000 7f551507b640 1 --2- 172.21.15.107:0/2325550182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f551006a5b0 0x7f551015be70 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:28.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.244+0000 7f551507b640 1 --2- 172.21.15.107:0/2325550182 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f55100c3990 0x7f551006e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:28.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.244+0000 7f551507b640 1 -- 172.21.15.107:0/2325550182 >> 172.21.15.107:0/2325550182 conn(0x7f551006b6c0 msgr2=0x7f551006bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:28.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.244+0000 7f551507b640 1 -- 172.21.15.107:0/2325550182 shutdown_connections 2023-12-04T05:42:28.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.244+0000 7f551507b640 1 -- 172.21.15.107:0/2325550182 wait complete. 2023-12-04T05:42:28.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.244+0000 7f551507b640 1 Processor -- start 2023-12-04T05:42:28.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.244+0000 7f551507b640 1 -- start start 2023-12-04T05:42:28.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.244+0000 7f551507b640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f551006a5b0 0x7f5510142440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:28.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.244+0000 7f551507b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55100c3990 0x7f55101f4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:28.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.244+0000 7f551507b640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f55100c3d70 0x7f55100c7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:28.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.244+0000 7f551507b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551014b390 con 0x7f55100c3990 2023-12-04T05:42:28.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.244+0000 7f551507b640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551014bbb0 con 0x7f55100c3d70 2023-12-04T05:42:28.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.244+0000 7f551507b640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f551014af50 con 0x7f551006a5b0 2023-12-04T05:42:28.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.244+0000 7f550e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55100c3990 0x7f55101f4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:28.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.248+0000 7f550e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55100c3990 0x7f55101f4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47410/0 (socket says 172.21.15.107:47410) 2023-12-04T05:42:28.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.248+0000 7f550e59c640 1 -- 172.21.15.107:0/36928627 learned_addr learned my addr 172.21.15.107:0/36928627 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:28.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.248+0000 7f550ed9d640 1 --2- 172.21.15.107:0/36928627 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f551006a5b0 0x7f5510142440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:28.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.248+0000 7f550f59e640 1 --2- 172.21.15.107:0/36928627 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f55100c3d70 0x7f55100c7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:28.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.248+0000 7f550e59c640 1 -- 172.21.15.107:0/36928627 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f551006a5b0 msgr2=0x7f5510142440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:28.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.248+0000 7f550e59c640 1 --2- 172.21.15.107:0/36928627 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f551006a5b0 0x7f5510142440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:28.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.248+0000 7f550e59c640 1 -- 172.21.15.107:0/36928627 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f55100c3d70 msgr2=0x7f55100c7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:28.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.248+0000 7f550e59c640 1 --2- 172.21.15.107:0/36928627 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f55100c3d70 0x7f55100c7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:28.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.248+0000 7f550e59c640 1 -- 172.21.15.107:0/36928627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f80094d0 con 0x7f55100c3990 2023-12-04T05:42:28.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.248+0000 7f550f59e640 1 --2- 172.21.15.107:0/36928627 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f55100c3d70 0x7f55100c7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:42:28.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.248+0000 7f550e59c640 1 --2- 172.21.15.107:0/36928627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55100c3990 0x7f55101f4430 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f54f80ae040 tx=0x7f54f8019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:28.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.248+0000 7f55077fe640 1 -- 172.21.15.107:0/36928627 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f80b4640 con 0x7f55100c3990 2023-12-04T05:42:28.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.248+0000 7f55077fe640 1 -- 172.21.15.107:0/36928627 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f80b47c0 con 0x7f55100c3990 2023-12-04T05:42:28.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.248+0000 7f55077fe640 1 -- 172.21.15.107:0/36928627 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f80b4b70 con 0x7f55100c3990 2023-12-04T05:42:28.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.248+0000 7f551507b640 1 -- 172.21.15.107:0/36928627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5510146c90 con 0x7f55100c3990 2023-12-04T05:42:28.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.248+0000 7f551507b640 1 -- 172.21.15.107:0/36928627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55101462a0 con 0x7f55100c3990 2023-12-04T05:42:28.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.248+0000 7f551507b640 1 -- 172.21.15.107:0/36928627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5510142160 con 0x7f55100c3990 2023-12-04T05:42:28.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.248+0000 7f55077fe640 1 -- 172.21.15.107:0/36928627 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f54f80b2070 con 0x7f55100c3990 2023-12-04T05:42:28.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.248+0000 7f55077fe640 1 --2- 172.21.15.107:0/36928627 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f54f00668d0 0x7f54f0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:28.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.248+0000 7f55077fe640 1 -- 172.21.15.107:0/36928627 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f54f811e4e0 con 0x7f55100c3990 2023-12-04T05:42:28.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.248+0000 7f550ed9d640 1 --2- 172.21.15.107:0/36928627 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f54f00668d0 0x7f54f0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:28.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.248+0000 7f550ed9d640 1 --2- 172.21.15.107:0/36928627 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f54f00668d0 0x7f54f0068d90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f550000a970 tx=0x7f5500008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:28.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.252+0000 7f55077fe640 1 -- 172.21.15.107:0/36928627 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5510142160 con 0x7f55100c3990 2023-12-04T05:42:28.434 INFO:tasks.workunit.client.0.smithi107.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2023-12-04T05:42:28.434 INFO:tasks.workunit.client.0.smithi107.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2023-12-04T05:42:28.434 INFO:tasks.workunit.client.0.smithi107.stderr:If you have problems, you may need to regenerate the build system entirely. 2023-12-04T05:42:28.434 INFO:tasks.workunit.client.0.smithi107.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2023-12-04T05:42:28.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.448+0000 7f551507b640 1 -- 172.21.15.107:0/36928627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5510142160 con 0x7f55100c3990 2023-12-04T05:42:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.448+0000 7f55077fe640 1 -- 172.21.15.107:0/36928627 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+5775 (secure 0 0 0) 0x7f5510142160 con 0x7f55100c3990 2023-12-04T05:42:28.455 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:42:28.455 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:42:25.729333+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:reconnect","state_seq":40,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:42:28.455 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 20 2023-12-04T05:42:28.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.452+0000 7f551507b640 1 -- 172.21.15.107:0/36928627 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f54f00668d0 msgr2=0x7f54f0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:28.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.452+0000 7f551507b640 1 --2- 172.21.15.107:0/36928627 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f54f00668d0 0x7f54f0068d90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f550000a970 tx=0x7f5500008040 comp rx=0 tx=0).stop 2023-12-04T05:42:28.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.452+0000 7f551507b640 1 -- 172.21.15.107:0/36928627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55100c3990 msgr2=0x7f55101f4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:28.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.452+0000 7f551507b640 1 --2- 172.21.15.107:0/36928627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55100c3990 0x7f55101f4430 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f54f80ae040 tx=0x7f54f8019210 comp rx=0 tx=0).stop 2023-12-04T05:42:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.452+0000 7f551507b640 1 -- 172.21.15.107:0/36928627 shutdown_connections 2023-12-04T05:42:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.452+0000 7f551507b640 1 --2- 172.21.15.107:0/36928627 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f54f00668d0 0x7f54f0068d90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.452+0000 7f551507b640 1 --2- 172.21.15.107:0/36928627 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f55100c3d70 0x7f55100c7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.452+0000 7f551507b640 1 --2- 172.21.15.107:0/36928627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f55100c3990 0x7f55101f4430 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.452+0000 7f551507b640 1 --2- 172.21.15.107:0/36928627 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f551006a5b0 0x7f5510142440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.452+0000 7f551507b640 1 -- 172.21.15.107:0/36928627 >> 172.21.15.107:0/36928627 conn(0x7f551006b6c0 msgr2=0x7f551006baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.452+0000 7f551507b640 1 -- 172.21.15.107:0/36928627 shutdown_connections 2023-12-04T05:42:28.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:28.452+0000 7f551507b640 1 -- 172.21.15.107:0/36928627 wait complete. 2023-12-04T05:42:28.475 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 20, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-12-04T05:39:59.032019+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 20, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4247": { "addr": "172.21.15.107:6839/993693883", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6838", "nonce": 993693883, "type": "v2" }, { "addr": "172.21.15.107:6839", "nonce": 993693883, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 18, "join_fscid": -1, "name": "a", "rank": 4, "state": "up:reconnect", "state_seq": 40 }, "gid_4260": { "addr": "172.21.15.107:6836/3444851515", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 3444851515, "type": "v2" }, { "addr": "172.21.15.107:6836", "nonce": 3444851515, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4260, "incarnation": 14, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 9 }, "gid_4263": { "addr": "172.21.15.107:6837/2210527429", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6835", "nonce": 2210527429, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 2210527429, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4263, "incarnation": 12, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 7 }, "gid_4279": { "addr": "172.21.15.195:6833/569093002", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6832", "nonce": 569093002, "type": "v2" }, { "addr": "172.21.15.195:6833", "nonce": 569093002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4279, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4288": { "addr": "172.21.15.195:6837/3412865477", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6835", "nonce": 3412865477, "type": "v2" }, { "addr": "172.21.15.195:6837", "nonce": 3412865477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 24, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-12-04T05:42:25.729333+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 4279, "mds_2": 4263, "mds_3": 4260, "mds_4": 4247 } } } ], "standbys": [] } 2023-12-04T05:42:29.411 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:4: warning: The macro `AC_FOREACH' is obsolete. 2023-12-04T05:42:29.411 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:4: You should run autoupdate. 2023-12-04T05:42:29.411 INFO:tasks.workunit.client.0.smithi107.stderr:./lib/autoconf/general.m4:191: AC_FOREACH is expanded from... 2023-12-04T05:42:29.411 INFO:tasks.workunit.client.0.smithi107.stderr:aclocal.m4:684: _AM_SET_OPTIONS is expanded from... 2023-12-04T05:42:29.411 INFO:tasks.workunit.client.0.smithi107.stderr:aclocal.m4:410: AM_INIT_AUTOMAKE is expanded from... 2023-12-04T05:42:29.411 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:4: the top level 2023-12-04T05:42:29.411 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:6: warning: The macro `AM_CONFIG_HEADER' is obsolete. 2023-12-04T05:42:29.411 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:6: You should run autoupdate. 2023-12-04T05:42:29.412 INFO:tasks.workunit.client.0.smithi107.stderr:aclocal.m4:383: AM_CONFIG_HEADER is expanded from... 2023-12-04T05:42:29.412 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:6: the top level 2023-12-04T05:42:29.412 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:10: warning: The macro `AC_AIX' is obsolete. 2023-12-04T05:42:29.412 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:10: You should run autoupdate. 2023-12-04T05:42:29.412 INFO:tasks.workunit.client.0.smithi107.stderr:./lib/autoconf/specific.m4:540: AC_AIX is expanded from... 2023-12-04T05:42:29.412 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:10: the top level 2023-12-04T05:42:29.412 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:36: warning: The macro `AC_HEADER_STDC' is obsolete. 2023-12-04T05:42:29.412 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:36: You should run autoupdate. 2023-12-04T05:42:29.412 INFO:tasks.workunit.client.0.smithi107.stderr:./lib/autoconf/headers.m4:704: AC_HEADER_STDC is expanded from... 2023-12-04T05:42:29.412 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:36: the top level 2023-12-04T05:42:29.412 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:42: warning: The macro `AC_HEADER_TIME' is obsolete. 2023-12-04T05:42:29.412 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:42: You should run autoupdate. 2023-12-04T05:42:29.412 INFO:tasks.workunit.client.0.smithi107.stderr:./lib/autoconf/headers.m4:743: AC_HEADER_TIME is expanded from... 2023-12-04T05:42:29.412 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:42: the top level 2023-12-04T05:42:29.413 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:45: warning: The macro `AC_FUNC_SETVBUF_REVERSED' is obsolete. Remove it and all references to SETVBUF_REVERSED. 2023-12-04T05:42:29.413 INFO:tasks.workunit.client.0.smithi107.stderr:./lib/autoconf/functions.m4:1766: AC_FUNC_SETVBUF_REVERSED is expanded from... 2023-12-04T05:42:29.413 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:45: the top level 2023-12-04T05:42:29.413 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:50: warning: AC_OUTPUT should be used without arguments. 2023-12-04T05:42:29.413 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:50: You should run autoupdate. 2023-12-04T05:42:29.586 INFO:tasks.workunit.client.0.smithi107.stdout:/bin/bash ./config.status --recheck 2023-12-04T05:42:29.746 INFO:tasks.workunit.client.0.smithi107.stdout:running CONFIG_SHELL=/bin/bash /bin/bash ./configure --no-create --no-recursion 2023-12-04T05:42:30.073 INFO:tasks.workunit.client.0.smithi107.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2023-12-04T05:42:30.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:42:30.243 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:42:30.476 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:42:30.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.600+0000 7f9cebc39640 1 Processor -- start 2023-12-04T05:42:30.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.600+0000 7f9cebc39640 1 -- start start 2023-12-04T05:42:30.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.600+0000 7f9cebc39640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9ce40b1b80 0x7f9ce40c5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:30.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.600+0000 7f9cebc39640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9ce4157d80 0x7f9ce40c5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:30.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.600+0000 7f9cebc39640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9ce4072680 0x7f9ce4072b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:30.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.600+0000 7f9cebc39640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ce40c7180 con 0x7f9ce4157d80 2023-12-04T05:42:30.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.600+0000 7f9cebc39640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ce40c3630 con 0x7f9ce40b1b80 2023-12-04T05:42:30.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.600+0000 7f9cebc39640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce40c4e00 con 0x7f9ce4072680 2023-12-04T05:42:30.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.600+0000 7f9ce91d4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9ce4157d80 0x7f9ce40c5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:30.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.600+0000 7f9ce91d4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9ce4157d80 0x7f9ce40c5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47414/0 (socket says 172.21.15.107:47414) 2023-12-04T05:42:30.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.600+0000 7f9ce91d4640 1 -- 172.21.15.107:0/3690807335 learned_addr learned my addr 172.21.15.107:0/3690807335 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:30.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.600+0000 7f9ce99d5640 1 --2- 172.21.15.107:0/3690807335 >> v2:172.21.15.195:3300/0 conn(0x7f9ce40b1b80 0x7f9ce40c5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:30.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.600+0000 7f9cea1d6640 1 --2- 172.21.15.107:0/3690807335 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9ce4072680 0x7f9ce4072b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:30.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.600+0000 7f9ce91d4640 1 -- 172.21.15.107:0/3690807335 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9ce4072680 msgr2=0x7f9ce4072b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:30.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.600+0000 7f9ce91d4640 1 --2- 172.21.15.107:0/3690807335 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9ce4072680 0x7f9ce4072b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:30.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.600+0000 7f9ce91d4640 1 -- 172.21.15.107:0/3690807335 >> v2:172.21.15.195:3300/0 conn(0x7f9ce40b1b80 msgr2=0x7f9ce40c5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:30.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.600+0000 7f9ce91d4640 1 --2- 172.21.15.107:0/3690807335 >> v2:172.21.15.195:3300/0 conn(0x7f9ce40b1b80 0x7f9ce40c5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:30.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.600+0000 7f9ce91d4640 1 -- 172.21.15.107:0/3690807335 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce40c30b0 con 0x7f9ce4157d80 2023-12-04T05:42:30.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.600+0000 7f9ce99d5640 1 --2- 172.21.15.107:0/3690807335 >> v2:172.21.15.195:3300/0 conn(0x7f9ce40b1b80 0x7f9ce40c5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:42:30.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.600+0000 7f9cea1d6640 1 --2- 172.21.15.107:0/3690807335 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9ce4072680 0x7f9ce4072b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:42:30.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9ce91d4640 1 --2- 172.21.15.107:0/3690807335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ce4157d80 0x7f9ce40c5290 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9ce40731a0 tx=0x7f9cd801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f699f89f92bf4bb1 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:30.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9ce89d3640 1 -- 172.21.15.107:0/3690807335 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd80ac630 con 0x7f9ce4157d80 2023-12-04T05:42:30.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9ce89d3640 1 -- 172.21.15.107:0/3690807335 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd80ac790 con 0x7f9ce4157d80 2023-12-04T05:42:30.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9ce89d3640 1 -- 172.21.15.107:0/3690807335 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd80acb20 con 0x7f9ce4157d80 2023-12-04T05:42:30.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cebc39640 1 -- 172.21.15.107:0/3690807335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ce4157d80 msgr2=0x7f9ce40c5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:30.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cebc39640 1 --2- 172.21.15.107:0/3690807335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ce4157d80 0x7f9ce40c5290 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9ce40731a0 tx=0x7f9cd801c500 comp rx=0 tx=0).stop 2023-12-04T05:42:30.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cebc39640 1 -- 172.21.15.107:0/3690807335 shutdown_connections 2023-12-04T05:42:30.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cebc39640 1 --2- 172.21.15.107:0/3690807335 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9ce4072680 0x7f9ce4072b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:30.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cebc39640 1 --2- 172.21.15.107:0/3690807335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ce4157d80 0x7f9ce40c5290 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:30.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cebc39640 1 --2- 172.21.15.107:0/3690807335 >> v2:172.21.15.195:3300/0 conn(0x7f9ce40b1b80 0x7f9ce40c5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:30.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cebc39640 1 -- 172.21.15.107:0/3690807335 >> 172.21.15.107:0/3690807335 conn(0x7f9ce406e710 msgr2=0x7f9ce40c8580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:30.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cebc39640 1 -- 172.21.15.107:0/3690807335 shutdown_connections 2023-12-04T05:42:30.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cebc39640 1 -- 172.21.15.107:0/3690807335 wait complete. 2023-12-04T05:42:30.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cebc39640 1 Processor -- start 2023-12-04T05:42:30.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cebc39640 1 -- start start 2023-12-04T05:42:30.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cebc39640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9ce4072680 0x7f9ce4157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:30.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cebc39640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9ce40b1b80 0x7f9ce414ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:30.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cebc39640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ce4157d80 0x7f9ce414e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:30.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cebc39640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce40c3bf0 con 0x7f9ce4157d80 2023-12-04T05:42:30.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cebc39640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce40c0740 con 0x7f9ce40b1b80 2023-12-04T05:42:30.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cebc39640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce40c2730 con 0x7f9ce4072680 2023-12-04T05:42:30.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cea1d6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ce4157d80 0x7f9ce414e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:30.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cea1d6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ce4157d80 0x7f9ce414e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47420/0 (socket says 172.21.15.107:47420) 2023-12-04T05:42:30.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cea1d6640 1 -- 172.21.15.107:0/2648340579 learned_addr learned my addr 172.21.15.107:0/2648340579 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:30.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9ce91d4640 1 --2- 172.21.15.107:0/2648340579 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9ce40b1b80 0x7f9ce414ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:30.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9ce99d5640 1 --2- 172.21.15.107:0/2648340579 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9ce4072680 0x7f9ce4157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:30.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cea1d6640 1 -- 172.21.15.107:0/2648340579 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9ce4072680 msgr2=0x7f9ce4157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:30.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cea1d6640 1 --2- 172.21.15.107:0/2648340579 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9ce4072680 0x7f9ce4157290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:30.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cea1d6640 1 -- 172.21.15.107:0/2648340579 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9ce40b1b80 msgr2=0x7f9ce414ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:30.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cea1d6640 1 --2- 172.21.15.107:0/2648340579 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9ce40b1b80 0x7f9ce414ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:30.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cea1d6640 1 -- 172.21.15.107:0/2648340579 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd80094d0 con 0x7f9ce4157d80 2023-12-04T05:42:30.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cea1d6640 1 --2- 172.21.15.107:0/2648340579 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ce4157d80 0x7f9ce414e310 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00c820 tx=0x7f9cdc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:30.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9ce27fc640 1 -- 172.21.15.107:0/2648340579 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc007a30 con 0x7f9ce4157d80 2023-12-04T05:42:30.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9ce27fc640 1 -- 172.21.15.107:0/2648340579 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cdc007b90 con 0x7f9ce4157d80 2023-12-04T05:42:30.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cebc39640 1 -- 172.21.15.107:0/2648340579 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce40c1e90 con 0x7f9ce4157d80 2023-12-04T05:42:30.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9ce27fc640 1 -- 172.21.15.107:0/2648340579 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc00ce10 con 0x7f9ce4157d80 2023-12-04T05:42:30.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.604+0000 7f9cebc39640 1 -- 172.21.15.107:0/2648340579 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce40c2450 con 0x7f9ce4157d80 2023-12-04T05:42:30.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.608+0000 7f9cebc39640 1 -- 172.21.15.107:0/2648340579 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce4067ec0 con 0x7f9ce4157d80 2023-12-04T05:42:30.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.608+0000 7f9ce27fc640 1 -- 172.21.15.107:0/2648340579 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9cdc024480 con 0x7f9ce4157d80 2023-12-04T05:42:30.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.608+0000 7f9ce27fc640 1 --2- 172.21.15.107:0/2648340579 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9cd00668d0 0x7f9cd0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:30.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.608+0000 7f9ce27fc640 1 -- 172.21.15.107:0/2648340579 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f9cdc084aa0 con 0x7f9ce4157d80 2023-12-04T05:42:30.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.608+0000 7f9ce99d5640 1 --2- 172.21.15.107:0/2648340579 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9cd00668d0 0x7f9cd0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:30.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.608+0000 7f9ce99d5640 1 --2- 172.21.15.107:0/2648340579 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9cd00668d0 0x7f9cd0068d90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4008f80 tx=0x7f9cd40084d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:30.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.612+0000 7f9ce27fc640 1 -- 172.21.15.107:0/2648340579 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9cdc051d20 con 0x7f9ce4157d80 2023-12-04T05:42:30.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.800+0000 7f9cebc39640 1 -- 172.21.15.107:0/2648340579 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ce40c4570 con 0x7f9ce4157d80 2023-12-04T05:42:30.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.800+0000 7f9ce27fc640 1 -- 172.21.15.107:0/2648340579 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+5775 (secure 0 0 0) 0x7f9cdc01f020 con 0x7f9ce4157d80 2023-12-04T05:42:30.808 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:42:30.808 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:42:25.729333+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:reconnect","state_seq":40,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:42:30.808 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 20 2023-12-04T05:42:30.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.804+0000 7f9cebc39640 1 -- 172.21.15.107:0/2648340579 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9cd00668d0 msgr2=0x7f9cd0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:30.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.804+0000 7f9cebc39640 1 --2- 172.21.15.107:0/2648340579 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9cd00668d0 0x7f9cd0068d90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4008f80 tx=0x7f9cd40084d0 comp rx=0 tx=0).stop 2023-12-04T05:42:30.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.804+0000 7f9cebc39640 1 -- 172.21.15.107:0/2648340579 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ce4157d80 msgr2=0x7f9ce414e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:30.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.804+0000 7f9cebc39640 1 --2- 172.21.15.107:0/2648340579 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ce4157d80 0x7f9ce414e310 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00c820 tx=0x7f9cdc00cce0 comp rx=0 tx=0).stop 2023-12-04T05:42:30.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.804+0000 7f9cebc39640 1 -- 172.21.15.107:0/2648340579 shutdown_connections 2023-12-04T05:42:30.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.804+0000 7f9cebc39640 1 --2- 172.21.15.107:0/2648340579 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9cd00668d0 0x7f9cd0068d90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:30.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.804+0000 7f9cebc39640 1 --2- 172.21.15.107:0/2648340579 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ce4157d80 0x7f9ce414e310 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:30.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.804+0000 7f9cebc39640 1 --2- 172.21.15.107:0/2648340579 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9ce40b1b80 0x7f9ce414ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:30.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.804+0000 7f9cebc39640 1 --2- 172.21.15.107:0/2648340579 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9ce4072680 0x7f9ce4157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:30.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.804+0000 7f9cebc39640 1 -- 172.21.15.107:0/2648340579 >> 172.21.15.107:0/2648340579 conn(0x7f9ce406e710 msgr2=0x7f9ce40c8050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:30.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.804+0000 7f9cebc39640 1 -- 172.21.15.107:0/2648340579 shutdown_connections 2023-12-04T05:42:30.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:30.804+0000 7f9cebc39640 1 -- 172.21.15.107:0/2648340579 wait complete. 2023-12-04T05:42:30.828 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:42:31.098 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether build environment is sane... yes 2023-12-04T05:42:31.115 INFO:tasks.workunit.client.0.smithi107.stdout:checking for a race-free mkdir -p... /usr/bin/mkdir -p 2023-12-04T05:42:31.116 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gawk... gawk 2023-12-04T05:42:31.157 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether make sets $(MAKE)... yes 2023-12-04T05:42:31.228 INFO:tasks.workunit.client.0.smithi107.stdout:checking build system type... x86_64-unknown-linux-gnu 2023-12-04T05:42:31.229 INFO:tasks.workunit.client.0.smithi107.stdout:checking host system type... x86_64-unknown-linux-gnu 2023-12-04T05:42:31.267 INFO:tasks.workunit.client.0.smithi107.stdout:checking for style of include used by make... GNU 2023-12-04T05:42:31.287 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc... gcc 2023-12-04T05:42:31.584 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether the C compiler works... yes 2023-12-04T05:42:31.584 INFO:tasks.workunit.client.0.smithi107.stdout:checking for C compiler default output file name... a.out 2023-12-04T05:42:31.696 INFO:tasks.workunit.client.0.smithi107.stdout:checking for suffix of executables... 2023-12-04T05:42:31.803 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether we are cross compiling... no 2023-12-04T05:42:31.912 INFO:tasks.workunit.client.0.smithi107.stdout:checking for suffix of object files... o 2023-12-04T05:42:32.002 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether the compiler supports GNU C... yes 2023-12-04T05:42:32.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:42:32.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.956+0000 7f6d8c2a0640 1 Processor -- start 2023-12-04T05:42:32.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.956+0000 7f6d8c2a0640 1 -- start start 2023-12-04T05:42:32.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.956+0000 7f6d8c2a0640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6d84154060 0x7f6d84159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:32.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.956+0000 7f6d8c2a0640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6d840c6140 0x7f6d840c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:32.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.956+0000 7f6d8c2a0640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8415ac80 con 0x7f6d840c6de0 2023-12-04T05:42:32.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.956+0000 7f6d8c2a0640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d84070790 con 0x7f6d840c6140 2023-12-04T05:42:32.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.956+0000 7f6d8c2a0640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d84154440 con 0x7f6d84154060 2023-12-04T05:42:32.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.956+0000 7f6d8a03c640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6d84154060 0x7f6d84159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:32.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.956+0000 7f6d8a03c640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6d84154060 0x7f6d84159ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.107:36774/0 (socket says 172.21.15.107:36774) 2023-12-04T05:42:32.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.956+0000 7f6d8a03c640 1 -- 172.21.15.107:0/1575425025 learned_addr learned my addr 172.21.15.107:0/1575425025 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:32.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.956+0000 7f6d8983b640 1 --2- 172.21.15.107:0/1575425025 >> v2:172.21.15.195:3300/0 conn(0x7f6d840c6140 0x7f6d840c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:32.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.956+0000 7f6d8a03c640 1 -- 172.21.15.107:0/1575425025 >> v2:172.21.15.195:3300/0 conn(0x7f6d840c6140 msgr2=0x7f6d840c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:32.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.956+0000 7f6d8a03c640 1 --2- 172.21.15.107:0/1575425025 >> v2:172.21.15.195:3300/0 conn(0x7f6d840c6140 0x7f6d840c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:32.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.956+0000 7f6d8a03c640 1 -- 172.21.15.107:0/1575425025 >> v1:172.21.15.195:6789/0 conn(0x7f6d840c6de0 legacy=0x7f6d84070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:32.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.956+0000 7f6d8a03c640 1 -- 172.21.15.107:0/1575425025 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d840cc390 con 0x7f6d84154060 2023-12-04T05:42:32.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.956+0000 7f6d8983b640 1 --2- 172.21.15.107:0/1575425025 >> v2:172.21.15.195:3300/0 conn(0x7f6d840c6140 0x7f6d840c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T05:42:32.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.960+0000 7f6d8a03c640 1 --2- 172.21.15.107:0/1575425025 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6d84154060 0x7f6d84159ce0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6d80009870 tx=0x7f6d8001c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dbb36cb0529707b server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:32.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.960+0000 7f6d8903a640 1 -- 172.21.15.107:0/1575425025 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d800a9400 con 0x7f6d84154060 2023-12-04T05:42:32.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.960+0000 7f6d8903a640 1 -- 172.21.15.107:0/1575425025 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d800a9560 con 0x7f6d84154060 2023-12-04T05:42:32.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.960+0000 7f6d8903a640 1 -- 172.21.15.107:0/1575425025 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d800a98f0 con 0x7f6d84154060 2023-12-04T05:42:32.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.960+0000 7f6d8c2a0640 1 -- 172.21.15.107:0/1575425025 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6d84154060 msgr2=0x7f6d84159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:32.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.960+0000 7f6d8c2a0640 1 --2- 172.21.15.107:0/1575425025 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6d84154060 0x7f6d84159ce0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6d80009870 tx=0x7f6d8001c540 comp rx=0 tx=0).stop 2023-12-04T05:42:32.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.960+0000 7f6d8c2a0640 1 -- 172.21.15.107:0/1575425025 shutdown_connections 2023-12-04T05:42:32.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8c2a0640 1 --2- 172.21.15.107:0/1575425025 >> v2:172.21.15.195:3300/0 conn(0x7f6d840c6140 0x7f6d840c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:32.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8c2a0640 1 --2- 172.21.15.107:0/1575425025 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6d84154060 0x7f6d84159ce0 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6d80009870 tx=0x7f6d8001c540 comp rx=0 tx=0).stop 2023-12-04T05:42:32.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8c2a0640 1 -- 172.21.15.107:0/1575425025 >> 172.21.15.107:0/1575425025 conn(0x7f6d8406c5a0 msgr2=0x7f6d8406dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:32.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8c2a0640 1 -- 172.21.15.107:0/1575425025 shutdown_connections 2023-12-04T05:42:32.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8c2a0640 1 -- 172.21.15.107:0/1575425025 wait complete. 2023-12-04T05:42:32.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8c2a0640 1 Processor -- start 2023-12-04T05:42:32.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8c2a0640 1 -- start start 2023-12-04T05:42:32.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8c2a0640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6d840c6140 0x7f6d841f2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:32.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8c2a0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d840c6de0 0x7f6d841f4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:32.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8c2a0640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6d84159650 0x7f6d84153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8c2a0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d841516c0 con 0x7f6d840c6de0 2023-12-04T05:42:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8c2a0640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d841f47a0 con 0x7f6d84159650 2023-12-04T05:42:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8c2a0640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d841518f0 con 0x7f6d840c6140 2023-12-04T05:42:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8983b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d840c6de0 0x7f6d841f4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8983b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d840c6de0 0x7f6d841f4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47426/0 (socket says 172.21.15.107:47426) 2023-12-04T05:42:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8983b640 1 -- 172.21.15.107:0/2786541997 learned_addr learned my addr 172.21.15.107:0/2786541997 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8a03c640 1 --2- 172.21.15.107:0/2786541997 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6d840c6140 0x7f6d841f2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8a83d640 1 --2- 172.21.15.107:0/2786541997 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6d84159650 0x7f6d84153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8983b640 1 -- 172.21.15.107:0/2786541997 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6d840c6140 msgr2=0x7f6d841f2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8983b640 1 --2- 172.21.15.107:0/2786541997 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6d840c6140 0x7f6d841f2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8983b640 1 -- 172.21.15.107:0/2786541997 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6d84159650 msgr2=0x7f6d84153400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8983b640 1 --2- 172.21.15.107:0/2786541997 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6d84159650 0x7f6d84153400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8983b640 1 -- 172.21.15.107:0/2786541997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d7c009a70 con 0x7f6d840c6de0 2023-12-04T05:42:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8a83d640 1 --2- 172.21.15.107:0/2786541997 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6d84159650 0x7f6d84153400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:42:32.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8983b640 1 --2- 172.21.15.107:0/2786541997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d840c6de0 0x7f6d841f4260 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c00efc0 tx=0x7f6d7c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:32.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d7affd640 1 -- 172.21.15.107:0/2786541997 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c014680 con 0x7f6d840c6de0 2023-12-04T05:42:32.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d7affd640 1 -- 172.21.15.107:0/2786541997 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d7c00f040 con 0x7f6d840c6de0 2023-12-04T05:42:32.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8c2a0640 1 -- 172.21.15.107:0/2786541997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d800094d0 con 0x7f6d840c6de0 2023-12-04T05:42:32.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d7affd640 1 -- 172.21.15.107:0/2786541997 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c014a00 con 0x7f6d840c6de0 2023-12-04T05:42:32.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.964+0000 7f6d8c2a0640 1 -- 172.21.15.107:0/2786541997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d84151eb0 con 0x7f6d840c6de0 2023-12-04T05:42:32.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.968+0000 7f6d8c2a0640 1 -- 172.21.15.107:0/2786541997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d84067870 con 0x7f6d840c6de0 2023-12-04T05:42:32.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.968+0000 7f6d7affd640 1 -- 172.21.15.107:0/2786541997 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f6d7c014b60 con 0x7f6d840c6de0 2023-12-04T05:42:32.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.968+0000 7f6d7affd640 1 --2- 172.21.15.107:0/2786541997 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6d680668d0 0x7f6d68068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:32.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.968+0000 7f6d7affd640 1 -- 172.21.15.107:0/2786541997 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f6d7c085540 con 0x7f6d840c6de0 2023-12-04T05:42:32.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.968+0000 7f6d8a03c640 1 --2- 172.21.15.107:0/2786541997 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6d680668d0 0x7f6d68068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:32.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.968+0000 7f6d8a03c640 1 --2- 172.21.15.107:0/2786541997 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6d680668d0 0x7f6d68068d90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6d800ae000 tx=0x7f6d800b3040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:32.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:32.972+0000 7f6d7affd640 1 -- 172.21.15.107:0/2786541997 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6d7c0526b0 con 0x7f6d840c6de0 2023-12-04T05:42:33.038 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether gcc accepts -g... yes 2023-12-04T05:42:33.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:33.152+0000 7f6d8c2a0640 1 -- 172.21.15.107:0/2786541997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d840c8e30 con 0x7f6d840c6de0 2023-12-04T05:42:33.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:33.152+0000 7f6d7affd640 1 -- 172.21.15.107:0/2786541997 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+5775 (secure 0 0 0) 0x7f6d7c055cd0 con 0x7f6d840c6de0 2023-12-04T05:42:33.160 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:42:33.160 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:42:25.729333+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:reconnect","state_seq":40,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:42:33.161 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 20 2023-12-04T05:42:33.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:33.156+0000 7f6d8c2a0640 1 -- 172.21.15.107:0/2786541997 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6d680668d0 msgr2=0x7f6d68068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:33.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:33.156+0000 7f6d8c2a0640 1 --2- 172.21.15.107:0/2786541997 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6d680668d0 0x7f6d68068d90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6d800ae000 tx=0x7f6d800b3040 comp rx=0 tx=0).stop 2023-12-04T05:42:33.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:33.156+0000 7f6d8c2a0640 1 -- 172.21.15.107:0/2786541997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d840c6de0 msgr2=0x7f6d841f4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:33.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:33.156+0000 7f6d8c2a0640 1 --2- 172.21.15.107:0/2786541997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d840c6de0 0x7f6d841f4260 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c00efc0 tx=0x7f6d7c00c490 comp rx=0 tx=0).stop 2023-12-04T05:42:33.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:33.156+0000 7f6d8c2a0640 1 -- 172.21.15.107:0/2786541997 shutdown_connections 2023-12-04T05:42:33.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:33.156+0000 7f6d8c2a0640 1 --2- 172.21.15.107:0/2786541997 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6d680668d0 0x7f6d68068d90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:33.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:33.156+0000 7f6d8c2a0640 1 --2- 172.21.15.107:0/2786541997 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6d84159650 0x7f6d84153400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:33.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:33.156+0000 7f6d8c2a0640 1 --2- 172.21.15.107:0/2786541997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d840c6de0 0x7f6d841f4260 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:33.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:33.156+0000 7f6d8c2a0640 1 --2- 172.21.15.107:0/2786541997 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6d840c6140 0x7f6d841f2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:33.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:33.156+0000 7f6d8c2a0640 1 -- 172.21.15.107:0/2786541997 >> 172.21.15.107:0/2786541997 conn(0x7f6d8406c5a0 msgr2=0x7f6d84070c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:33.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:33.156+0000 7f6d8c2a0640 1 -- 172.21.15.107:0/2786541997 shutdown_connections 2023-12-04T05:42:33.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:33.156+0000 7f6d8c2a0640 1 -- 172.21.15.107:0/2786541997 wait complete. 2023-12-04T05:42:33.177 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:42:34.036 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc option to enable C11 features... none needed 2023-12-04T05:42:34.430 INFO:tasks.workunit.client.0.smithi107.stdout:checking dependency style of gcc... gcc3 2023-12-04T05:42:34.538 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stdio.h... yes 2023-12-04T05:42:34.649 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stdlib.h... yes 2023-12-04T05:42:34.758 INFO:tasks.workunit.client.0.smithi107.stdout:checking for string.h... yes 2023-12-04T05:42:34.865 INFO:tasks.workunit.client.0.smithi107.stdout:checking for inttypes.h... yes 2023-12-04T05:42:34.978 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stdint.h... yes 2023-12-04T05:42:35.083 INFO:tasks.workunit.client.0.smithi107.stdout:checking for strings.h... yes 2023-12-04T05:42:35.178 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:42:35.192 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/stat.h... yes 2023-12-04T05:42:35.297 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/types.h... yes 2023-12-04T05:42:35.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.300+0000 7feb4d843640 1 Processor -- start 2023-12-04T05:42:35.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.300+0000 7feb4d843640 1 -- start start 2023-12-04T05:42:35.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.300+0000 7feb4d843640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7feb4806dc50 0x7feb4806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:35.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb4d843640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feb4815d350 0x7feb4815d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:35.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb4d843640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb480c8380 con 0x7feb48153d00 2023-12-04T05:42:35.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb4d843640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb4815dcc0 con 0x7feb4815d350 2023-12-04T05:42:35.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb4d843640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb480c8640 con 0x7feb4806dc50 2023-12-04T05:42:35.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb477fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feb4815d350 0x7feb4815d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:35.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb477fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feb4815d350 0x7feb4815d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47436/0 (socket says 172.21.15.107:47436) 2023-12-04T05:42:35.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb477fe640 1 -- 172.21.15.107:0/1241653392 learned_addr learned my addr 172.21.15.107:0/1241653392 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:35.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb467fc640 1 --2- 172.21.15.107:0/1241653392 >> v2:172.21.15.195:3300/0 conn(0x7feb4806dc50 0x7feb4806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:35.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb477fe640 1 -- 172.21.15.107:0/1241653392 >> v2:172.21.15.195:3300/0 conn(0x7feb4806dc50 msgr2=0x7feb4806e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:35.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb477fe640 1 --2- 172.21.15.107:0/1241653392 >> v2:172.21.15.195:3300/0 conn(0x7feb4806dc50 0x7feb4806e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:35.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb477fe640 1 -- 172.21.15.107:0/1241653392 >> v1:172.21.15.195:6789/0 conn(0x7feb48153d00 legacy=0x7feb48066450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:35.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb477fe640 1 -- 172.21.15.107:0/1241653392 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4815a0b0 con 0x7feb4815d350 2023-12-04T05:42:35.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb477fe640 1 --2- 172.21.15.107:0/1241653392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb4815d350 0x7feb4815d780 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7feb480ccef0 tx=0x7feb4001f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31de5498af8f3408 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:35.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb45ffb640 1 -- 172.21.15.107:0/1241653392 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb400af400 con 0x7feb4815d350 2023-12-04T05:42:35.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb45ffb640 1 -- 172.21.15.107:0/1241653392 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb400ab050 con 0x7feb4815d350 2023-12-04T05:42:35.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb45ffb640 1 -- 172.21.15.107:0/1241653392 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb400af780 con 0x7feb4815d350 2023-12-04T05:42:35.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb4d843640 1 -- 172.21.15.107:0/1241653392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb4815d350 msgr2=0x7feb4815d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:35.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb4d843640 1 --2- 172.21.15.107:0/1241653392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb4815d350 0x7feb4815d780 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7feb480ccef0 tx=0x7feb4001f180 comp rx=0 tx=0).stop 2023-12-04T05:42:35.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb4d843640 1 -- 172.21.15.107:0/1241653392 shutdown_connections 2023-12-04T05:42:35.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb4d843640 1 --2- 172.21.15.107:0/1241653392 >> v2:172.21.15.195:3300/0 conn(0x7feb4806dc50 0x7feb4806e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:35.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb4d843640 1 --2- 172.21.15.107:0/1241653392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb4815d350 0x7feb4815d780 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:35.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb4d843640 1 -- 172.21.15.107:0/1241653392 >> 172.21.15.107:0/1241653392 conn(0x7feb4806ada0 msgr2=0x7feb480cbe30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:35.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb4d843640 1 -- 172.21.15.107:0/1241653392 shutdown_connections 2023-12-04T05:42:35.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.304+0000 7feb4d843640 1 -- 172.21.15.107:0/1241653392 wait complete. 2023-12-04T05:42:35.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb4d843640 1 Processor -- start 2023-12-04T05:42:35.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb4d843640 1 -- start start 2023-12-04T05:42:35.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb4d843640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb4806dc50 0x7feb480c6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:35.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb4d843640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb48153d00 0x7feb4815c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:35.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb4d843640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feb4815d350 0x7feb480c9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:35.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb4d843640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb48158050 con 0x7feb48153d00 2023-12-04T05:42:35.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb4d843640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb48159950 con 0x7feb4806dc50 2023-12-04T05:42:35.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb4d843640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb48157a20 con 0x7feb4815d350 2023-12-04T05:42:35.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb467fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb48153d00 0x7feb4815c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:35.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb467fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb48153d00 0x7feb4815c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47450/0 (socket says 172.21.15.107:47450) 2023-12-04T05:42:35.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb467fc640 1 -- 172.21.15.107:0/607063126 learned_addr learned my addr 172.21.15.107:0/607063126 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:35.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb46ffd640 1 --2- 172.21.15.107:0/607063126 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb4806dc50 0x7feb480c6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:35.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb467fc640 1 -- 172.21.15.107:0/607063126 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feb4815d350 msgr2=0x7feb480c9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:35.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb467fc640 1 --2- 172.21.15.107:0/607063126 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feb4815d350 0x7feb480c9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:35.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb467fc640 1 -- 172.21.15.107:0/607063126 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb4806dc50 msgr2=0x7feb480c6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:35.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb467fc640 1 --2- 172.21.15.107:0/607063126 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb4806dc50 0x7feb480c6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:35.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb467fc640 1 -- 172.21.15.107:0/607063126 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb40009b30 con 0x7feb48153d00 2023-12-04T05:42:35.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb467fc640 1 --2- 172.21.15.107:0/607063126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb48153d00 0x7feb4815c510 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7feb3c00d830 tx=0x7feb3c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:35.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb2ffff640 1 -- 172.21.15.107:0/607063126 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c00b660 con 0x7feb48153d00 2023-12-04T05:42:35.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb2ffff640 1 -- 172.21.15.107:0/607063126 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3c00b7c0 con 0x7feb48153d00 2023-12-04T05:42:35.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb4d843640 1 -- 172.21.15.107:0/607063126 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb480cb6c0 con 0x7feb48153d00 2023-12-04T05:42:35.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb2ffff640 1 -- 172.21.15.107:0/607063126 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c00bb30 con 0x7feb48153d00 2023-12-04T05:42:35.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb4d843640 1 -- 172.21.15.107:0/607063126 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb48157c70 con 0x7feb48153d00 2023-12-04T05:42:35.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.308+0000 7feb4d843640 1 -- 172.21.15.107:0/607063126 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb480cbbe0 con 0x7feb48153d00 2023-12-04T05:42:35.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.312+0000 7feb2ffff640 1 -- 172.21.15.107:0/607063126 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7feb3c01e460 con 0x7feb48153d00 2023-12-04T05:42:35.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.312+0000 7feb2ffff640 1 --2- 172.21.15.107:0/607063126 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7feb340668d0 0x7feb34068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:35.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.312+0000 7feb2ffff640 1 -- 172.21.15.107:0/607063126 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7feb3c084bb0 con 0x7feb48153d00 2023-12-04T05:42:35.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.312+0000 7feb46ffd640 1 --2- 172.21.15.107:0/607063126 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7feb340668d0 0x7feb34068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:35.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.312+0000 7feb46ffd640 1 --2- 172.21.15.107:0/607063126 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7feb340668d0 0x7feb34068d90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7feb38004e30 tx=0x7feb38009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:35.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.316+0000 7feb2ffff640 1 -- 172.21.15.107:0/607063126 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7feb3c051db0 con 0x7feb48153d00 2023-12-04T05:42:35.415 INFO:tasks.workunit.client.0.smithi107.stdout:checking for unistd.h... yes 2023-12-04T05:42:35.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.512+0000 7feb4d843640 1 -- 172.21.15.107:0/607063126 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb48067ec0 con 0x7feb48153d00 2023-12-04T05:42:35.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.512+0000 7feb2ffff640 1 -- 172.21.15.107:0/607063126 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+5775 (secure 0 0 0) 0x7feb3c0553d0 con 0x7feb48153d00 2023-12-04T05:42:35.520 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:42:35.520 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:42:25.729333+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:reconnect","state_seq":40,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:42:35.520 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 20 2023-12-04T05:42:35.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.516+0000 7feb4d843640 1 -- 172.21.15.107:0/607063126 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7feb340668d0 msgr2=0x7feb34068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:35.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.516+0000 7feb4d843640 1 --2- 172.21.15.107:0/607063126 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7feb340668d0 0x7feb34068d90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7feb38004e30 tx=0x7feb38009040 comp rx=0 tx=0).stop 2023-12-04T05:42:35.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.516+0000 7feb4d843640 1 -- 172.21.15.107:0/607063126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb48153d00 msgr2=0x7feb4815c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:35.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.516+0000 7feb4d843640 1 --2- 172.21.15.107:0/607063126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb48153d00 0x7feb4815c510 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7feb3c00d830 tx=0x7feb3c00dcf0 comp rx=0 tx=0).stop 2023-12-04T05:42:35.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.516+0000 7feb4d843640 1 -- 172.21.15.107:0/607063126 shutdown_connections 2023-12-04T05:42:35.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.516+0000 7feb4d843640 1 --2- 172.21.15.107:0/607063126 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7feb340668d0 0x7feb34068d90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:35.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.516+0000 7feb4d843640 1 --2- 172.21.15.107:0/607063126 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feb4815d350 0x7feb480c9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:35.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.516+0000 7feb4d843640 1 --2- 172.21.15.107:0/607063126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb48153d00 0x7feb4815c510 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:35.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.516+0000 7feb4d843640 1 --2- 172.21.15.107:0/607063126 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feb4806dc50 0x7feb480c6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:35.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.516+0000 7feb4d843640 1 -- 172.21.15.107:0/607063126 >> 172.21.15.107:0/607063126 conn(0x7feb4806ada0 msgr2=0x7feb48073a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:35.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.516+0000 7feb4d843640 1 -- 172.21.15.107:0/607063126 shutdown_connections 2023-12-04T05:42:35.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:35.516+0000 7feb4d843640 1 -- 172.21.15.107:0/607063126 wait complete. 2023-12-04T05:42:35.539 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:42:35.549 INFO:tasks.workunit.client.0.smithi107.stdout:checking for wchar.h... yes 2023-12-04T05:42:36.438 INFO:tasks.workunit.client.0.smithi107.stdout:checking for minix/config.h... no 2023-12-04T05:42:36.569 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/time.h... yes 2023-12-04T05:42:37.540 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:42:37.645 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2023-12-04T05:42:37.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.664+0000 7fab3486f640 1 Processor -- start 2023-12-04T05:42:37.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.664+0000 7fab3486f640 1 -- start start 2023-12-04T05:42:37.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.664+0000 7fab3486f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fab30150dc0 0x7fab3015a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:37.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.664+0000 7fab3486f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fab30071080 0x7fab3015ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:37.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.664+0000 7fab3486f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab3015a390 con 0x7fab30156b70 2023-12-04T05:42:37.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.664+0000 7fab3486f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab30157ba0 con 0x7fab30071080 2023-12-04T05:42:37.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.664+0000 7fab3486f640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab30157040 con 0x7fab30150dc0 2023-12-04T05:42:37.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.664+0000 7fab2ed9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fab30071080 0x7fab3015ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:37.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.664+0000 7fab2ed9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fab30071080 0x7fab3015ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50710/0 (socket says 172.21.15.107:50710) 2023-12-04T05:42:37.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.664+0000 7fab2ed9d640 1 -- 172.21.15.107:0/1197062827 learned_addr learned my addr 172.21.15.107:0/1197062827 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:37.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.664+0000 7fab2e59c640 1 --2- 172.21.15.107:0/1197062827 >> v2:172.21.15.195:3300/0 conn(0x7fab30150dc0 0x7fab3015a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:37.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.664+0000 7fab2ed9d640 1 -- 172.21.15.107:0/1197062827 >> v2:172.21.15.195:3300/0 conn(0x7fab30150dc0 msgr2=0x7fab3015a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:37.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.664+0000 7fab2ed9d640 1 --2- 172.21.15.107:0/1197062827 >> v2:172.21.15.195:3300/0 conn(0x7fab30150dc0 0x7fab3015a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:37.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.664+0000 7fab2ed9d640 1 -- 172.21.15.107:0/1197062827 >> v1:172.21.15.195:6789/0 conn(0x7fab30156b70 legacy=0x7fab30071740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:37.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.664+0000 7fab2ed9d640 1 -- 172.21.15.107:0/1197062827 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab30157300 con 0x7fab30071080 2023-12-04T05:42:37.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.664+0000 7fab2e59c640 1 --2- 172.21.15.107:0/1197062827 >> v2:172.21.15.195:3300/0 conn(0x7fab30150dc0 0x7fab3015a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:42:37.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.668+0000 7fab2ed9d640 1 --2- 172.21.15.107:0/1197062827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab30071080 0x7fab3015ad60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fab300c9fa0 tx=0x7fab2801c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c34d4bbf821e3cad server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:37.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.668+0000 7fab2d59a640 1 -- 172.21.15.107:0/1197062827 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab280a5850 con 0x7fab30071080 2023-12-04T05:42:37.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.668+0000 7fab2d59a640 1 -- 172.21.15.107:0/1197062827 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab280a59b0 con 0x7fab30071080 2023-12-04T05:42:37.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.668+0000 7fab2d59a640 1 -- 172.21.15.107:0/1197062827 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab280a5d40 con 0x7fab30071080 2023-12-04T05:42:37.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.668+0000 7fab3486f640 1 -- 172.21.15.107:0/1197062827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab30071080 msgr2=0x7fab3015ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:37.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.668+0000 7fab3486f640 1 --2- 172.21.15.107:0/1197062827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab30071080 0x7fab3015ad60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fab300c9fa0 tx=0x7fab2801c990 comp rx=0 tx=0).stop 2023-12-04T05:42:37.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.668+0000 7fab3486f640 1 -- 172.21.15.107:0/1197062827 shutdown_connections 2023-12-04T05:42:37.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.668+0000 7fab3486f640 1 --2- 172.21.15.107:0/1197062827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab30071080 0x7fab3015ad60 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:37.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.668+0000 7fab3486f640 1 --2- 172.21.15.107:0/1197062827 >> v2:172.21.15.195:3300/0 conn(0x7fab30150dc0 0x7fab3015a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:37.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.668+0000 7fab3486f640 1 -- 172.21.15.107:0/1197062827 >> 172.21.15.107:0/1197062827 conn(0x7fab300cc490 msgr2=0x7fab300cc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:37.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.668+0000 7fab3486f640 1 -- 172.21.15.107:0/1197062827 shutdown_connections 2023-12-04T05:42:37.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.668+0000 7fab3486f640 1 -- 172.21.15.107:0/1197062827 wait complete. 2023-12-04T05:42:37.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.668+0000 7fab3486f640 1 Processor -- start 2023-12-04T05:42:37.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.668+0000 7fab3486f640 1 -- start start 2023-12-04T05:42:37.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab3486f640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fab30071080 0x7fab301380a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:37.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab3486f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fab30150dc0 0x7fab300cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:37.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab3486f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab30156b70 0x7fab300cba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:37.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab3486f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab301f6680 con 0x7fab30156b70 2023-12-04T05:42:37.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab3486f640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab301f2d40 con 0x7fab30071080 2023-12-04T05:42:37.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab3486f640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab301385e0 con 0x7fab30150dc0 2023-12-04T05:42:37.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab2ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab30156b70 0x7fab300cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:37.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab2ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab30156b70 0x7fab300cba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50726/0 (socket says 172.21.15.107:50726) 2023-12-04T05:42:37.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab2ed9d640 1 -- 172.21.15.107:0/606845436 learned_addr learned my addr 172.21.15.107:0/606845436 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:37.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab2e59c640 1 --2- 172.21.15.107:0/606845436 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fab30071080 0x7fab301380a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:37.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab2dd9b640 1 --2- 172.21.15.107:0/606845436 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fab30150dc0 0x7fab300cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:37.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab2ed9d640 1 -- 172.21.15.107:0/606845436 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fab30150dc0 msgr2=0x7fab300cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:37.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab2ed9d640 1 --2- 172.21.15.107:0/606845436 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fab30150dc0 0x7fab300cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:37.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab2ed9d640 1 -- 172.21.15.107:0/606845436 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fab30071080 msgr2=0x7fab301380a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:37.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab2ed9d640 1 --2- 172.21.15.107:0/606845436 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fab30071080 0x7fab301380a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:37.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab2ed9d640 1 -- 172.21.15.107:0/606845436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab28009b30 con 0x7fab30156b70 2023-12-04T05:42:37.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab2e59c640 1 --2- 172.21.15.107:0/606845436 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fab30071080 0x7fab301380a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:42:37.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab2ed9d640 1 --2- 172.21.15.107:0/606845436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab30156b70 0x7fab300cba20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fab280af040 tx=0x7fab28019730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:37.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab1effd640 1 -- 172.21.15.107:0/606845436 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab280b8670 con 0x7fab30156b70 2023-12-04T05:42:37.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab1effd640 1 -- 172.21.15.107:0/606845436 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab280b87d0 con 0x7fab30156b70 2023-12-04T05:42:37.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab1effd640 1 -- 172.21.15.107:0/606845436 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab280b8b80 con 0x7fab30156b70 2023-12-04T05:42:37.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab3486f640 1 -- 172.21.15.107:0/606845436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab301568d0 con 0x7fab30156b70 2023-12-04T05:42:37.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab3486f640 1 -- 172.21.15.107:0/606845436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab301f3230 con 0x7fab30156b70 2023-12-04T05:42:37.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab3486f640 1 -- 172.21.15.107:0/606845436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab30067ec0 con 0x7fab30156b70 2023-12-04T05:42:37.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.672+0000 7fab1effd640 1 -- 172.21.15.107:0/606845436 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fab280b8ce0 con 0x7fab30156b70 2023-12-04T05:42:37.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.676+0000 7fab1effd640 1 --2- 172.21.15.107:0/606845436 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fab100668d0 0x7fab10068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:37.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.676+0000 7fab1effd640 1 -- 172.21.15.107:0/606845436 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fab2811fa40 con 0x7fab30156b70 2023-12-04T05:42:37.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.676+0000 7fab2e59c640 1 --2- 172.21.15.107:0/606845436 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fab100668d0 0x7fab10068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:37.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.676+0000 7fab2e59c640 1 --2- 172.21.15.107:0/606845436 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fab100668d0 0x7fab10068d90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fab20006fd0 tx=0x7fab20008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:37.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.680+0000 7fab1effd640 1 -- 172.21.15.107:0/606845436 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fab28121e60 con 0x7fab30156b70 2023-12-04T05:42:37.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.868+0000 7fab3486f640 1 -- 172.21.15.107:0/606845436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab3013f020 con 0x7fab30156b70 2023-12-04T05:42:37.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.872+0000 7fab1effd640 1 -- 172.21.15.107:0/606845436 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5772 (secure 0 0 0) 0x7fab280ecc40 con 0x7fab30156b70 2023-12-04T05:42:37.877 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:42:37.877 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:42:34.831250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:rejoin","state_seq":43,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:42:37.877 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 21 2023-12-04T05:42:37.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.872+0000 7fab3486f640 1 -- 172.21.15.107:0/606845436 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fab100668d0 msgr2=0x7fab10068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:37.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.872+0000 7fab3486f640 1 --2- 172.21.15.107:0/606845436 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fab100668d0 0x7fab10068d90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fab20006fd0 tx=0x7fab20008040 comp rx=0 tx=0).stop 2023-12-04T05:42:37.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.872+0000 7fab3486f640 1 -- 172.21.15.107:0/606845436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab30156b70 msgr2=0x7fab300cba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:37.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.872+0000 7fab3486f640 1 --2- 172.21.15.107:0/606845436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab30156b70 0x7fab300cba20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fab280af040 tx=0x7fab28019730 comp rx=0 tx=0).stop 2023-12-04T05:42:37.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.872+0000 7fab3486f640 1 -- 172.21.15.107:0/606845436 shutdown_connections 2023-12-04T05:42:37.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.872+0000 7fab3486f640 1 --2- 172.21.15.107:0/606845436 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fab100668d0 0x7fab10068d90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:37.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.872+0000 7fab3486f640 1 --2- 172.21.15.107:0/606845436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab30156b70 0x7fab300cba20 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:37.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.872+0000 7fab3486f640 1 --2- 172.21.15.107:0/606845436 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fab30150dc0 0x7fab300cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:37.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.872+0000 7fab3486f640 1 --2- 172.21.15.107:0/606845436 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fab30071080 0x7fab301380a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:37.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.872+0000 7fab3486f640 1 -- 172.21.15.107:0/606845436 >> 172.21.15.107:0/606845436 conn(0x7fab300cc490 msgr2=0x7fab30160de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:37.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.872+0000 7fab3486f640 1 -- 172.21.15.107:0/606845436 shutdown_connections 2023-12-04T05:42:37.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:37.872+0000 7fab3486f640 1 -- 172.21.15.107:0/606845436 wait complete. 2023-12-04T05:42:37.896 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:42:38.576 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether _XOPEN_SOURCE should be defined... no 2023-12-04T05:42:38.617 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc... (cached) gcc 2023-12-04T05:42:38.841 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether the compiler supports GNU C... (cached) yes 2023-12-04T05:42:38.842 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether gcc accepts -g... (cached) yes 2023-12-04T05:42:38.842 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc option to enable C11 features... (cached) none needed 2023-12-04T05:42:38.843 INFO:tasks.workunit.client.0.smithi107.stdout:checking dependency style of gcc... (cached) gcc3 2023-12-04T05:42:39.001 INFO:tasks.workunit.client.0.smithi107.stdout:checking for main in -lm... yes 2023-12-04T05:42:39.165 INFO:tasks.workunit.client.0.smithi107.stdout:checking for main in -lpthread... yes 2023-12-04T05:42:39.176 INFO:tasks.workunit.client.0.smithi107.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2023-12-04T05:42:39.178 INFO:tasks.workunit.client.0.smithi107.stdout:checking for egrep... /usr/bin/grep -E 2023-12-04T05:42:39.282 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2023-12-04T05:42:39.413 INFO:tasks.workunit.client.0.smithi107.stdout:checking for pthread.h... yes 2023-12-04T05:42:39.540 INFO:tasks.workunit.client.0.smithi107.stdout:checking for fcntl.h... yes 2023-12-04T05:42:39.661 INFO:tasks.workunit.client.0.smithi107.stdout:checking for limits.h... yes 2023-12-04T05:42:39.670 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stdint.h... (cached) yes 2023-12-04T05:42:39.671 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/time.h... (cached) yes 2023-12-04T05:42:39.672 INFO:tasks.workunit.client.0.smithi107.stdout:checking for unistd.h... (cached) yes 2023-12-04T05:42:39.792 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/vfs.h... yes 2023-12-04T05:42:39.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:42:40.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bc21c8640 1 Processor -- start 2023-12-04T05:42:40.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bc21c8640 1 -- start start 2023-12-04T05:42:40.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bc21c8640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4bbc154020 0x7f4bbc15a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:40.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bc21c8640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4bbc0c4c10 0x7f4bbc0c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:40.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bc21c8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc15ad50 con 0x7f4bbc0d01c0 2023-12-04T05:42:40.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bc21c8640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bbc0ca230 con 0x7f4bbc154020 2023-12-04T05:42:40.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bc21c8640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc0c6770 con 0x7f4bbc0c4c10 2023-12-04T05:42:40.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bbaffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4bbc0d01c0 0x7f4bbc071ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54174/0 (socket says 172.21.15.107:54174) 2023-12-04T05:42:40.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bbaffd640 1 -- 172.21.15.107:0/1723131007 learned_addr learned my addr 172.21.15.107:0/1723131007 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:42:40.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bbb7fe640 1 --2- 172.21.15.107:0/1723131007 >> v2:172.21.15.195:3300/0 conn(0x7f4bbc154020 0x7f4bbc15a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:40.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bbbfff640 1 --2- 172.21.15.107:0/1723131007 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4bbc0c4c10 0x7f4bbc0c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:40.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bba7fc640 1 -- 172.21.15.107:0/1723131007 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134189522 0 0) 0x7f4bbc15ad50 con 0x7f4bbc0d01c0 2023-12-04T05:42:40.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bba7fc640 1 -- 172.21.15.107:0/1723131007 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ba8003530 con 0x7f4bbc0d01c0 2023-12-04T05:42:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bba7fc640 1 -- 172.21.15.107:0/1723131007 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3256386074 0 0) 0x7f4ba8003530 con 0x7f4bbc0d01c0 2023-12-04T05:42:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bba7fc640 1 -- 172.21.15.107:0/1723131007 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4bbc0c4c10 msgr2=0x7f4bbc0c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bba7fc640 1 --2- 172.21.15.107:0/1723131007 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4bbc0c4c10 0x7f4bbc0c5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bba7fc640 1 -- 172.21.15.107:0/1723131007 >> v2:172.21.15.195:3300/0 conn(0x7f4bbc154020 msgr2=0x7f4bbc15a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bba7fc640 1 --2- 172.21.15.107:0/1723131007 >> v2:172.21.15.195:3300/0 conn(0x7f4bbc154020 0x7f4bbc15a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bba7fc640 1 -- 172.21.15.107:0/1723131007 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc0c55a0 con 0x7f4bbc0d01c0 2023-12-04T05:42:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bba7fc640 1 -- 172.21.15.107:0/1723131007 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f4bac0016a0 con 0x7f4bbc0d01c0 2023-12-04T05:42:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bba7fc640 1 -- 172.21.15.107:0/1723131007 >> v1:172.21.15.107:6789/0 conn(0x7f4bbc0d01c0 legacy=0x7f4bbc071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bba7fc640 1 --2- 172.21.15.107:0/1723131007 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4bbc0c4c10 0x7f4ba8003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bba7fc640 1 --2- 172.21.15.107:0/1723131007 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba8003f00 0x7f4ba80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bba7fc640 1 --2- 172.21.15.107:0/1723131007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ba8004d10 0x7f4ba8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bba7fc640 1 -- 172.21.15.107:0/1723131007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba800b990 con 0x7f4ba8004d10 2023-12-04T05:42:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bba7fc640 1 -- 172.21.15.107:0/1723131007 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba800bad0 con 0x7f4ba8003f00 2023-12-04T05:42:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bba7fc640 1 -- 172.21.15.107:0/1723131007 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba800bc40 con 0x7f4bbc0c4c10 2023-12-04T05:42:40.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bbbfff640 1 --2- 172.21.15.107:0/1723131007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ba8004d10 0x7f4ba8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:40.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bbb7fe640 1 --2- 172.21.15.107:0/1723131007 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4bbc0c4c10 0x7f4ba8003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:40.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bbaffd640 1 --2- 172.21.15.107:0/1723131007 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba8003f00 0x7f4ba80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:40.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bbbfff640 1 -- 172.21.15.107:0/1723131007 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4bbc0c4c10 msgr2=0x7f4ba8003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:40.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bbbfff640 1 --2- 172.21.15.107:0/1723131007 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4bbc0c4c10 0x7f4ba8003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:40.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bbbfff640 1 -- 172.21.15.107:0/1723131007 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba8003f00 msgr2=0x7f4ba80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:40.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bbbfff640 1 --2- 172.21.15.107:0/1723131007 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ba8003f00 0x7f4ba80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:40.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.024+0000 7f4bbbfff640 1 -- 172.21.15.107:0/1723131007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba8003530 con 0x7f4ba8004d10 2023-12-04T05:42:40.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.028+0000 7f4bbbfff640 1 --2- 172.21.15.107:0/1723131007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ba8004d10 0x7f4ba8005190 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4009fc0 tx=0x7f4bb401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=539084e29bc7ad49 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:40.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.028+0000 7f4bba7fc640 1 -- 172.21.15.107:0/1723131007 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb40a5c70 con 0x7f4ba8004d10 2023-12-04T05:42:40.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.028+0000 7f4bba7fc640 1 -- 172.21.15.107:0/1723131007 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb400f040 con 0x7f4ba8004d10 2023-12-04T05:42:40.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.028+0000 7f4bba7fc640 1 -- 172.21.15.107:0/1723131007 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb40ad440 con 0x7f4ba8004d10 2023-12-04T05:42:40.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.028+0000 7f4bc21c8640 1 -- 172.21.15.107:0/1723131007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ba8004d10 msgr2=0x7f4ba8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:40.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.028+0000 7f4bc21c8640 1 --2- 172.21.15.107:0/1723131007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ba8004d10 0x7f4ba8005190 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4009fc0 tx=0x7f4bb401cc60 comp rx=0 tx=0).stop 2023-12-04T05:42:40.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.028+0000 7f4bbbfff640 1 -- 172.21.15.107:0/1723131007 reap_dead start 2023-12-04T05:42:40.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.028+0000 7f4bc21c8640 1 -- 172.21.15.107:0/1723131007 shutdown_connections 2023-12-04T05:42:40.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.028+0000 7f4bc21c8640 1 -- 172.21.15.107:0/1723131007 >> 172.21.15.107:0/1723131007 conn(0x7f4bbc06c6c0 msgr2=0x7f4bbc0cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:40.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.028+0000 7f4bc21c8640 1 -- 172.21.15.107:0/1723131007 shutdown_connections 2023-12-04T05:42:40.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.028+0000 7f4bc21c8640 1 -- 172.21.15.107:0/1723131007 wait complete. 2023-12-04T05:42:40.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.028+0000 7f4bc21c8640 1 Processor -- start 2023-12-04T05:42:40.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.028+0000 7f4bc21c8640 1 -- start start 2023-12-04T05:42:40.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.028+0000 7f4bc21c8640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4ba8004d10 0x7f4bbc14eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:40.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bc21c8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bbc0cf410 0x7f4bbc1f25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:40.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bc21c8640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bbc0c7700 0x7f4bbc0c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:40.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bc21c8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc0c9130 con 0x7f4bbc0cf410 2023-12-04T05:42:40.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bc21c8640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc0c80c0 con 0x7f4bbc0c7700 2023-12-04T05:42:40.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bc21c8640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc0c6f20 con 0x7f4ba8004d10 2023-12-04T05:42:40.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bbaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bbc0cf410 0x7f4bbc1f25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:40.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bbaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bbc0cf410 0x7f4bbc1f25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50740/0 (socket says 172.21.15.107:50740) 2023-12-04T05:42:40.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bbaffd640 1 -- 172.21.15.107:0/2994466717 learned_addr learned my addr 172.21.15.107:0/2994466717 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:40.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bbbfff640 1 --2- 172.21.15.107:0/2994466717 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bbc0c7700 0x7f4bbc0c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:40.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bbb7fe640 1 --2- 172.21.15.107:0/2994466717 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4ba8004d10 0x7f4bbc14eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:40.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bbaffd640 1 -- 172.21.15.107:0/2994466717 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4ba8004d10 msgr2=0x7f4bbc14eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:40.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bbaffd640 1 --2- 172.21.15.107:0/2994466717 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4ba8004d10 0x7f4bbc14eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:40.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bbaffd640 1 -- 172.21.15.107:0/2994466717 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bbc0c7700 msgr2=0x7f4bbc0c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:40.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bbaffd640 1 --2- 172.21.15.107:0/2994466717 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bbc0c7700 0x7f4bbc0c7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:40.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bbaffd640 1 -- 172.21.15.107:0/2994466717 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc14f040 con 0x7f4bbc0cf410 2023-12-04T05:42:40.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bbaffd640 1 --2- 172.21.15.107:0/2994466717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bbc0cf410 0x7f4bbc1f25e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4bac00e7b0 tx=0x7f4bac00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:40.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bb3fff640 1 -- 172.21.15.107:0/2994466717 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac00c8d0 con 0x7f4bbc0cf410 2023-12-04T05:42:40.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bb3fff640 1 -- 172.21.15.107:0/2994466717 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bac00ca50 con 0x7f4bbc0cf410 2023-12-04T05:42:40.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bb3fff640 1 -- 172.21.15.107:0/2994466717 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac00ce00 con 0x7f4bbc0cf410 2023-12-04T05:42:40.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bc21c8640 1 -- 172.21.15.107:0/2994466717 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bbc0c6a90 con 0x7f4bbc0cf410 2023-12-04T05:42:40.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bc21c8640 1 -- 172.21.15.107:0/2994466717 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bbc0c9340 con 0x7f4bbc0cf410 2023-12-04T05:42:40.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bc21c8640 1 -- 172.21.15.107:0/2994466717 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bbc0c7090 con 0x7f4bbc0cf410 2023-12-04T05:42:40.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bb3fff640 1 -- 172.21.15.107:0/2994466717 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4bac01d460 con 0x7f4bbc0cf410 2023-12-04T05:42:40.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bb3fff640 1 --2- 172.21.15.107:0/2994466717 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4b9c066900 0x7f4b9c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:40.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bbb7fe640 1 --2- 172.21.15.107:0/2994466717 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4b9c066900 0x7f4b9c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:40.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bb3fff640 1 -- 172.21.15.107:0/2994466717 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f4bac084420 con 0x7f4bbc0cf410 2023-12-04T05:42:40.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.032+0000 7f4bbb7fe640 1 --2- 172.21.15.107:0/2994466717 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4b9c066900 0x7f4b9c068dc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4009ad0 tx=0x7f4ba4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:40.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.036+0000 7f4bb3fff640 1 -- 172.21.15.107:0/2994466717 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4bac051590 con 0x7f4bbc0cf410 2023-12-04T05:42:40.164 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/limits.h... no 2023-12-04T05:42:40.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.236+0000 7f4bc21c8640 1 -- 172.21.15.107:0/2994466717 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bbc067870 con 0x7f4bbc0cf410 2023-12-04T05:42:40.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.236+0000 7f4bb3fff640 1 -- 172.21.15.107:0/2994466717 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5772 (secure 0 0 0) 0x7f4bac054bb0 con 0x7f4bbc0cf410 2023-12-04T05:42:40.242 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:42:40.242 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:42:34.831250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:rejoin","state_seq":43,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:42:40.242 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 21 2023-12-04T05:42:40.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.236+0000 7f4bc21c8640 1 -- 172.21.15.107:0/2994466717 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4b9c066900 msgr2=0x7f4b9c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:40.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.236+0000 7f4bc21c8640 1 --2- 172.21.15.107:0/2994466717 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4b9c066900 0x7f4b9c068dc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4009ad0 tx=0x7f4ba4008040 comp rx=0 tx=0).stop 2023-12-04T05:42:40.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.236+0000 7f4bc21c8640 1 -- 172.21.15.107:0/2994466717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bbc0cf410 msgr2=0x7f4bbc1f25e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:40.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.236+0000 7f4bc21c8640 1 --2- 172.21.15.107:0/2994466717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bbc0cf410 0x7f4bbc1f25e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4bac00e7b0 tx=0x7f4bac00ec70 comp rx=0 tx=0).stop 2023-12-04T05:42:40.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.240+0000 7f4bc21c8640 1 -- 172.21.15.107:0/2994466717 shutdown_connections 2023-12-04T05:42:40.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.240+0000 7f4bc21c8640 1 --2- 172.21.15.107:0/2994466717 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4b9c066900 0x7f4b9c068dc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:40.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.240+0000 7f4bc21c8640 1 --2- 172.21.15.107:0/2994466717 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bbc0c7700 0x7f4bbc0c7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:40.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.240+0000 7f4bc21c8640 1 --2- 172.21.15.107:0/2994466717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bbc0cf410 0x7f4bbc1f25e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:40.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.240+0000 7f4bc21c8640 1 --2- 172.21.15.107:0/2994466717 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4ba8004d10 0x7f4bbc14eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:40.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.240+0000 7f4bc21c8640 1 -- 172.21.15.107:0/2994466717 >> 172.21.15.107:0/2994466717 conn(0x7f4bbc06c6c0 msgr2=0x7f4bbc152a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:40.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.240+0000 7f4bc21c8640 1 -- 172.21.15.107:0/2994466717 shutdown_connections 2023-12-04T05:42:40.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:40.240+0000 7f4bc21c8640 1 -- 172.21.15.107:0/2994466717 wait complete. 2023-12-04T05:42:40.250 INFO:tasks.workunit.client.0.smithi107.stdout:checking for an ANSI C-conforming const... yes 2023-12-04T05:42:40.262 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 21, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-12-04T05:39:59.032019+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 21, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4247": { "addr": "172.21.15.107:6839/993693883", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6838", "nonce": 993693883, "type": "v2" }, { "addr": "172.21.15.107:6839", "nonce": 993693883, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 18, "join_fscid": -1, "name": "a", "rank": 4, "state": "up:rejoin", "state_seq": 43 }, "gid_4260": { "addr": "172.21.15.107:6836/3444851515", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 3444851515, "type": "v2" }, { "addr": "172.21.15.107:6836", "nonce": 3444851515, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4260, "incarnation": 14, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 9 }, "gid_4263": { "addr": "172.21.15.107:6837/2210527429", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6835", "nonce": 2210527429, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 2210527429, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4263, "incarnation": 12, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 7 }, "gid_4279": { "addr": "172.21.15.195:6833/569093002", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6832", "nonce": 569093002, "type": "v2" }, { "addr": "172.21.15.195:6833", "nonce": 569093002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4279, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4288": { "addr": "172.21.15.195:6837/3412865477", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6835", "nonce": 3412865477, "type": "v2" }, { "addr": "172.21.15.195:6837", "nonce": 3412865477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 24, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-12-04T05:42:34.831250+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 4279, "mds_2": 4263, "mds_3": 4260, "mds_4": 4247 } } } ], "standbys": [] } 2023-12-04T05:42:40.394 INFO:tasks.workunit.client.0.smithi107.stdout:checking for system... yes 2023-12-04T05:42:40.548 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gettimeofday... yes 2023-12-04T05:42:40.700 INFO:tasks.workunit.client.0.smithi107.stdout:checking for mkdir... yes 2023-12-04T05:42:41.634 INFO:tasks.workunit.client.0.smithi107.stdout:checking for strerror... yes 2023-12-04T05:42:42.264 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:42:42.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.388+0000 7fb8ed261640 1 Processor -- start 2023-12-04T05:42:42.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.388+0000 7fb8ed261640 1 -- start start 2023-12-04T05:42:42.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8ed261640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb8e80d01c0 0x7fb8e8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:42.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8ed261640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e815d710 con 0x7fb8e8072220 2023-12-04T05:42:42.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8ed261640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e815d9f0 con 0x7fb8e8154020 2023-12-04T05:42:42.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8ed261640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e80ca230 con 0x7fb8e80d01c0 2023-12-04T05:42:42.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e77fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb8e8072220 0x7fb8e80c4c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54186/0 (socket says 172.21.15.107:54186) 2023-12-04T05:42:42.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e77fe640 1 -- 172.21.15.107:0/1666741745 learned_addr learned my addr 172.21.15.107:0/1666741745 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:42:42.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e67fc640 1 --2- 172.21.15.107:0/1666741745 >> v2:172.21.15.195:3300/0 conn(0x7fb8e80d01c0 0x7fb8e8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:42.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e5ffb640 1 -- 172.21.15.107:0/1666741745 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045655296 0 0) 0x7fb8e815d710 con 0x7fb8e8072220 2023-12-04T05:42:42.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e5ffb640 1 -- 172.21.15.107:0/1666741745 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8d0003530 con 0x7fb8e8072220 2023-12-04T05:42:42.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e5ffb640 1 -- 172.21.15.107:0/1666741745 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3521745988 0 0) 0x7fb8d0003530 con 0x7fb8e8072220 2023-12-04T05:42:42.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e5ffb640 1 -- 172.21.15.107:0/1666741745 >> v2:172.21.15.195:3300/0 conn(0x7fb8e80d01c0 msgr2=0x7fb8e8071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:42.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e5ffb640 1 --2- 172.21.15.107:0/1666741745 >> v2:172.21.15.195:3300/0 conn(0x7fb8e80d01c0 0x7fb8e8071ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:42.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e5ffb640 1 -- 172.21.15.107:0/1666741745 >> v1:172.21.15.195:6789/0 conn(0x7fb8e8154020 legacy=0x7fb8e815a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:42.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e5ffb640 1 -- 172.21.15.107:0/1666741745 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e815dcd0 con 0x7fb8e8072220 2023-12-04T05:42:42.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e5ffb640 1 -- 172.21.15.107:0/1666741745 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7fb8e0001a00 con 0x7fb8e8072220 2023-12-04T05:42:42.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e5ffb640 1 -- 172.21.15.107:0/1666741745 >> v1:172.21.15.107:6789/0 conn(0x7fb8e8072220 legacy=0x7fb8e80c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:42.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e5ffb640 1 --2- 172.21.15.107:0/1666741745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8d0003980 0x7fb8d0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:42.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e5ffb640 1 --2- 172.21.15.107:0/1666741745 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb8d00047d0 0x7fb8d0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:42.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e5ffb640 1 --2- 172.21.15.107:0/1666741745 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb8d0005190 0x7fb8d000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:42.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e5ffb640 1 -- 172.21.15.107:0/1666741745 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d000c010 con 0x7fb8d0003980 2023-12-04T05:42:42.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e5ffb640 1 -- 172.21.15.107:0/1666741745 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d000c180 con 0x7fb8d00047d0 2023-12-04T05:42:42.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e5ffb640 1 -- 172.21.15.107:0/1666741745 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d000c340 con 0x7fb8d0005190 2023-12-04T05:42:42.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e6ffd640 1 --2- 172.21.15.107:0/1666741745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8d0003980 0x7fb8d0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:42.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e67fc640 1 --2- 172.21.15.107:0/1666741745 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb8d00047d0 0x7fb8d0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:42.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e77fe640 1 --2- 172.21.15.107:0/1666741745 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb8d0005190 0x7fb8d000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:42.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e6ffd640 1 -- 172.21.15.107:0/1666741745 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb8d0005190 msgr2=0x7fb8d000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:42.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e6ffd640 1 --2- 172.21.15.107:0/1666741745 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb8d0005190 0x7fb8d000b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:42.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e6ffd640 1 -- 172.21.15.107:0/1666741745 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb8d00047d0 msgr2=0x7fb8d0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:42.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e6ffd640 1 --2- 172.21.15.107:0/1666741745 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb8d00047d0 0x7fb8d0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:42.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e6ffd640 1 -- 172.21.15.107:0/1666741745 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d0003530 con 0x7fb8d0003980 2023-12-04T05:42:42.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e77fe640 1 -- 172.21.15.107:0/1666741745 reap_dead start 2023-12-04T05:42:42.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e6ffd640 1 --2- 172.21.15.107:0/1666741745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8d0003980 0x7fb8d0003da0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8001780 tx=0x7fb8d801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60064965796190ef server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:42.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e5ffb640 1 -- 172.21.15.107:0/1666741745 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d80052d0 con 0x7fb8d0003980 2023-12-04T05:42:42.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e5ffb640 1 -- 172.21.15.107:0/1666741745 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8d80a7050 con 0x7fb8d0003980 2023-12-04T05:42:42.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.392+0000 7fb8e5ffb640 1 -- 172.21.15.107:0/1666741745 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d8003600 con 0x7fb8d0003980 2023-12-04T05:42:42.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8ed261640 1 -- 172.21.15.107:0/1666741745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8d0003980 msgr2=0x7fb8d0003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:42.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8ed261640 1 --2- 172.21.15.107:0/1666741745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8d0003980 0x7fb8d0003da0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8001780 tx=0x7fb8d801c400 comp rx=0 tx=0).stop 2023-12-04T05:42:42.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8ed261640 1 -- 172.21.15.107:0/1666741745 shutdown_connections 2023-12-04T05:42:42.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8ed261640 1 --2- 172.21.15.107:0/1666741745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8d0003980 0x7fb8d0003da0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:42.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8ed261640 1 -- 172.21.15.107:0/1666741745 >> 172.21.15.107:0/1666741745 conn(0x7fb8e806c6c0 msgr2=0x7fb8e80cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:42.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8ed261640 1 -- 172.21.15.107:0/1666741745 shutdown_connections 2023-12-04T05:42:42.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8ed261640 1 -- 172.21.15.107:0/1666741745 wait complete. 2023-12-04T05:42:42.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8ed261640 1 Processor -- start 2023-12-04T05:42:42.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8ed261640 1 -- start start 2023-12-04T05:42:42.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8ed261640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb8d0003980 0x7fb8e8155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:42.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8ed261640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8e8152110 0x7fb8e81f0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:42.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8ed261640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb8e815c410 0x7fb8e8149f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:42.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8ed261640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e81f0560 con 0x7fb8e8152110 2023-12-04T05:42:42.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8ed261640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e81513e0 con 0x7fb8d0003980 2023-12-04T05:42:42.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8ed261640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e8151c80 con 0x7fb8e815c410 2023-12-04T05:42:42.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8e67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8e8152110 0x7fb8e81f0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:42.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8e67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8e8152110 0x7fb8e81f0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50764/0 (socket says 172.21.15.107:50764) 2023-12-04T05:42:42.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8e67fc640 1 -- 172.21.15.107:0/1434867842 learned_addr learned my addr 172.21.15.107:0/1434867842 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:42.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8e6ffd640 1 --2- 172.21.15.107:0/1434867842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb8d0003980 0x7fb8e8155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:42.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8e77fe640 1 --2- 172.21.15.107:0/1434867842 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb8e815c410 0x7fb8e8149f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:42.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8e67fc640 1 -- 172.21.15.107:0/1434867842 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb8e815c410 msgr2=0x7fb8e8149f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:42.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8e67fc640 1 --2- 172.21.15.107:0/1434867842 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb8e815c410 0x7fb8e8149f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:42.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8e67fc640 1 -- 172.21.15.107:0/1434867842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb8d0003980 msgr2=0x7fb8e8155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:42.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8e67fc640 1 --2- 172.21.15.107:0/1434867842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb8d0003980 0x7fb8e8155690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:42.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8e67fc640 1 -- 172.21.15.107:0/1434867842 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e8152590 con 0x7fb8e8152110 2023-12-04T05:42:42.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8e67fc640 1 --2- 172.21.15.107:0/1434867842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8e8152110 0x7fb8e81f0020 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc011500 tx=0x7fb8dc0119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:42.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8cf7fe640 1 -- 172.21.15.107:0/1434867842 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc015870 con 0x7fb8e8152110 2023-12-04T05:42:42.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8cf7fe640 1 -- 172.21.15.107:0/1434867842 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8dc0159d0 con 0x7fb8e8152110 2023-12-04T05:42:42.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8ed261640 1 -- 172.21.15.107:0/1434867842 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e8152bf0 con 0x7fb8e8152110 2023-12-04T05:42:42.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8cf7fe640 1 -- 172.21.15.107:0/1434867842 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc015d40 con 0x7fb8e8152110 2023-12-04T05:42:42.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.396+0000 7fb8ed261640 1 -- 172.21.15.107:0/1434867842 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e8151630 con 0x7fb8e8152110 2023-12-04T05:42:42.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.400+0000 7fb8ed261640 1 -- 172.21.15.107:0/1434867842 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e8067870 con 0x7fb8e8152110 2023-12-04T05:42:42.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.400+0000 7fb8cf7fe640 1 -- 172.21.15.107:0/1434867842 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fb8dc027430 con 0x7fb8e8152110 2023-12-04T05:42:42.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.400+0000 7fb8cf7fe640 1 --2- 172.21.15.107:0/1434867842 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb8d40668d0 0x7fb8d4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:42.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.400+0000 7fb8cf7fe640 1 -- 172.21.15.107:0/1434867842 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fb8dc084cf0 con 0x7fb8e8152110 2023-12-04T05:42:42.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.400+0000 7fb8e6ffd640 1 --2- 172.21.15.107:0/1434867842 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb8d40668d0 0x7fb8d4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:42.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.400+0000 7fb8e6ffd640 1 --2- 172.21.15.107:0/1434867842 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb8d40668d0 0x7fb8d4068d90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8004000 tx=0x7fb8d80011a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:42.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.404+0000 7fb8cf7fe640 1 -- 172.21.15.107:0/1434867842 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb8dc051e60 con 0x7fb8e8152110 2023-12-04T05:42:42.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.592+0000 7fb8ed261640 1 -- 172.21.15.107:0/1434867842 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8e80cfae0 con 0x7fb8e8152110 2023-12-04T05:42:42.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.592+0000 7fb8cf7fe640 1 -- 172.21.15.107:0/1434867842 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5772 (secure 0 0 0) 0x7fb8dc055480 con 0x7fb8e8152110 2023-12-04T05:42:42.597 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:42:42.597 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:42:40.730334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:active","state_seq":45,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:42:42.597 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 22 2023-12-04T05:42:42.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.592+0000 7fb8ed261640 1 -- 172.21.15.107:0/1434867842 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb8d40668d0 msgr2=0x7fb8d4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:42.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.592+0000 7fb8ed261640 1 --2- 172.21.15.107:0/1434867842 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb8d40668d0 0x7fb8d4068d90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8004000 tx=0x7fb8d80011a0 comp rx=0 tx=0).stop 2023-12-04T05:42:42.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.592+0000 7fb8ed261640 1 -- 172.21.15.107:0/1434867842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8e8152110 msgr2=0x7fb8e81f0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:42.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.592+0000 7fb8ed261640 1 --2- 172.21.15.107:0/1434867842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8e8152110 0x7fb8e81f0020 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc011500 tx=0x7fb8dc0119c0 comp rx=0 tx=0).stop 2023-12-04T05:42:42.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.592+0000 7fb8ed261640 1 -- 172.21.15.107:0/1434867842 shutdown_connections 2023-12-04T05:42:42.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.592+0000 7fb8ed261640 1 --2- 172.21.15.107:0/1434867842 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb8d40668d0 0x7fb8d4068d90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:42.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.592+0000 7fb8ed261640 1 --2- 172.21.15.107:0/1434867842 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb8e815c410 0x7fb8e8149f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:42.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.592+0000 7fb8ed261640 1 --2- 172.21.15.107:0/1434867842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8e8152110 0x7fb8e81f0020 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:42.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.592+0000 7fb8ed261640 1 --2- 172.21.15.107:0/1434867842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb8d0003980 0x7fb8e8155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:42.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.592+0000 7fb8ed261640 1 -- 172.21.15.107:0/1434867842 >> 172.21.15.107:0/1434867842 conn(0x7fb8e806c6c0 msgr2=0x7fb8e80cab00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:42.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.592+0000 7fb8ed261640 1 -- 172.21.15.107:0/1434867842 shutdown_connections 2023-12-04T05:42:42.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:42.592+0000 7fb8ed261640 1 -- 172.21.15.107:0/1434867842 wait complete. 2023-12-04T05:42:42.617 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=4, replacing gid=4250 2023-12-04T05:42:42.617 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.b 2023-12-04T05:42:42.717 INFO:tasks.workunit.client.0.smithi107.stdout:checking for open64... yes 2023-12-04T05:42:43.129 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stat64... yes 2023-12-04T05:42:43.292 INFO:tasks.workunit.client.0.smithi107.stdout:checking for fseeko64... yes 2023-12-04T05:42:43.451 INFO:tasks.workunit.client.0.smithi107.stdout:checking for lrand48_r... yes 2023-12-04T05:42:43.603 INFO:tasks.workunit.client.0.smithi107.stdout:checking for srand48_r... yes 2023-12-04T05:42:43.641 INFO:tasks.workunit.client.0.smithi107.stdout:configure: creating ./config.status 2023-12-04T05:42:43.825 INFO:tasks.workunit.client.0.smithi107.stdout: /bin/bash ./config.status 2023-12-04T05:42:43.988 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: creating Makefile 2023-12-04T05:42:44.035 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: creating config.h 2023-12-04T05:42:44.091 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: executing depfiles commands 2023-12-04T05:42:44.114 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: executing default commands 2023-12-04T05:42:44.230 INFO:tasks.workunit.client.0.smithi107.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2023-12-04T05:42:44.281 INFO:tasks.workunit.client.0.smithi107.stderr:autoheader: warning: autoconf input should be named 'configure.ac', not 'configure.in' 2023-12-04T05:42:44.374 INFO:tasks.workunit.client.0.smithi107.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2023-12-04T05:42:44.374 INFO:tasks.workunit.client.0.smithi107.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2023-12-04T05:42:44.374 INFO:tasks.workunit.client.0.smithi107.stderr:If you have problems, you may need to regenerate the build system entirely. 2023-12-04T05:42:44.374 INFO:tasks.workunit.client.0.smithi107.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2023-12-04T05:42:44.547 INFO:tasks.workunit.client.0.smithi107.stdout:rm -f stamp-h1 2023-12-04T05:42:44.559 INFO:tasks.workunit.client.0.smithi107.stdout:touch config.h.in 2023-12-04T05:42:44.570 INFO:tasks.workunit.client.0.smithi107.stdout:cd . && /bin/bash ./config.status config.h 2023-12-04T05:42:45.272 INFO:tasks.ceph.mds.f.smithi195.stderr:2023-12-04T05:42:45.269+0000 7f5bdadb8640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-12-04T05:42:45.661 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: creating config.h 2023-12-04T05:42:45.755 INFO:tasks.workunit.client.0.smithi107.stdout:make all-am 2023-12-04T05:42:45.794 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2023-12-04T05:42:45.794 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2023-12-04T05:42:46.047 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2023-12-04T05:42:46.081 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2023-12-04T05:42:47.072 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2023-12-04T05:42:47.102 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2023-12-04T05:42:48.292 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/main.Tpo .deps/main.Po 2023-12-04T05:42:48.317 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2023-12-04T05:42:48.647 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2023-12-04T05:42:48.670 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2023-12-04T05:42:49.369 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2023-12-04T05:42:49.396 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2023-12-04T05:42:50.526 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2023-12-04T05:42:50.554 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2023-12-04T05:42:50.617 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2023-12-04T05:42:50.618 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-04T05:42:50.618 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-04T05:42:50.620 INFO:tasks.ceph.mds.b:Started 2023-12-04T05:42:50.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:42:50.699 INFO:tasks.ceph.mds.b.smithi195.stdout:starting mds.b at 2023-12-04T05:42:50.710 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2023-12-04T05:42:50.724 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2023-12-04T05:42:50.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.748+0000 7f3536c8e640 1 Processor -- start 2023-12-04T05:42:50.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.748+0000 7f3536c8e640 1 -- start start 2023-12-04T05:42:50.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.752+0000 7f3536c8e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f353006a810 0x7f35300c6d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:50.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.752+0000 7f3536c8e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3530159f40 0x7f3530153da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:50.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.752+0000 7f3536c8e640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f353015a3d0 con 0x7f353006abf0 2023-12-04T05:42:50.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.752+0000 7f3536c8e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3530072680 con 0x7f3530159f40 2023-12-04T05:42:50.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.752+0000 7f3536c8e640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35300c8d00 con 0x7f353006a810 2023-12-04T05:42:50.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.752+0000 7f3527fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3530159f40 0x7f3530153da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:50.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.752+0000 7f3534a2a640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f353006a810 0x7f35300c6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:50.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.752+0000 7f3527fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3530159f40 0x7f3530153da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58136/0 (socket says 172.21.15.107:58136) 2023-12-04T05:42:50.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.752+0000 7f3527fff640 1 -- 172.21.15.107:0/762868421 learned_addr learned my addr 172.21.15.107:0/762868421 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:50.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.752+0000 7f3527fff640 1 -- 172.21.15.107:0/762868421 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f353006a810 msgr2=0x7f35300c6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:50.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.752+0000 7f3527fff640 1 --2- 172.21.15.107:0/762868421 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f353006a810 0x7f35300c6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:50.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.752+0000 7f3527fff640 1 -- 172.21.15.107:0/762868421 >> v1:172.21.15.195:6789/0 conn(0x7f353006abf0 legacy=0x7f3530071280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:50.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.752+0000 7f3527fff640 1 -- 172.21.15.107:0/762868421 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353015a6b0 con 0x7f3530159f40 2023-12-04T05:42:50.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.752+0000 7f3527fff640 1 --2- 172.21.15.107:0/762868421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3530159f40 0x7f3530153da0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3520009870 tx=0x7f352001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e1040c541d99325 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:50.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.752+0000 7f35277fe640 1 -- 172.21.15.107:0/762868421 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35200a9620 con 0x7f3530159f40 2023-12-04T05:42:50.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.752+0000 7f35277fe640 1 -- 172.21.15.107:0/762868421 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35200a9780 con 0x7f3530159f40 2023-12-04T05:42:50.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.752+0000 7f35277fe640 1 -- 172.21.15.107:0/762868421 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35200a9b10 con 0x7f3530159f40 2023-12-04T05:42:50.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3536c8e640 1 -- 172.21.15.107:0/762868421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3530159f40 msgr2=0x7f3530153da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:50.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3536c8e640 1 --2- 172.21.15.107:0/762868421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3530159f40 0x7f3530153da0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3520009870 tx=0x7f352001c540 comp rx=0 tx=0).stop 2023-12-04T05:42:50.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3536c8e640 1 -- 172.21.15.107:0/762868421 shutdown_connections 2023-12-04T05:42:50.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3536c8e640 1 --2- 172.21.15.107:0/762868421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3530159f40 0x7f3530153da0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:50.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3536c8e640 1 --2- 172.21.15.107:0/762868421 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f353006a810 0x7f35300c6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:50.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3536c8e640 1 -- 172.21.15.107:0/762868421 >> 172.21.15.107:0/762868421 conn(0x7f35300cc440 msgr2=0x7f35300cc850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:50.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3536c8e640 1 -- 172.21.15.107:0/762868421 shutdown_connections 2023-12-04T05:42:50.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3536c8e640 1 -- 172.21.15.107:0/762868421 wait complete. 2023-12-04T05:42:50.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3536c8e640 1 Processor -- start 2023-12-04T05:42:50.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3536c8e640 1 -- start start 2023-12-04T05:42:50.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3536c8e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f353006a810 0x7f35301f62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:50.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3536c8e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f353006abf0 0x7f353014f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:50.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3536c8e640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f353014ad30 0x7f353014bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:50.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3536c8e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353014f3d0 con 0x7f353006a810 2023-12-04T05:42:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3536c8e640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353014f080 con 0x7f353014ad30 2023-12-04T05:42:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3536c8e640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3530150ae0 con 0x7f353006abf0 2023-12-04T05:42:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3534a2a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f353006a810 0x7f35301f62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3534a2a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f353006a810 0x7f35301f62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58150/0 (socket says 172.21.15.107:58150) 2023-12-04T05:42:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3534a2a640 1 -- 172.21.15.107:0/1568656730 learned_addr learned my addr 172.21.15.107:0/1568656730 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f353522b640 1 --2- 172.21.15.107:0/1568656730 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f353014ad30 0x7f353014bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3527fff640 1 --2- 172.21.15.107:0/1568656730 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f353006abf0 0x7f353014f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3534a2a640 1 -- 172.21.15.107:0/1568656730 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f353006abf0 msgr2=0x7f353014f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3534a2a640 1 --2- 172.21.15.107:0/1568656730 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f353006abf0 0x7f353014f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3534a2a640 1 -- 172.21.15.107:0/1568656730 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f353014ad30 msgr2=0x7f353014bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3534a2a640 1 --2- 172.21.15.107:0/1568656730 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f353014ad30 0x7f353014bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3534a2a640 1 -- 172.21.15.107:0/1568656730 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35200094d0 con 0x7f353006a810 2023-12-04T05:42:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3534a2a640 1 --2- 172.21.15.107:0/1568656730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f353006a810 0x7f35301f62f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f352c00efc0 tx=0x7f352c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f35257fa640 1 -- 172.21.15.107:0/1568656730 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352c014480 con 0x7f353006a810 2023-12-04T05:42:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f35257fa640 1 -- 172.21.15.107:0/1568656730 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f352c00f040 con 0x7f353006a810 2023-12-04T05:42:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f35257fa640 1 -- 172.21.15.107:0/1568656730 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352c014800 con 0x7f353006a810 2023-12-04T05:42:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3536c8e640 1 -- 172.21.15.107:0/1568656730 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3530150dc0 con 0x7f353006a810 2023-12-04T05:42:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.756+0000 7f3536c8e640 1 -- 172.21.15.107:0/1568656730 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35301507d0 con 0x7f353006a810 2023-12-04T05:42:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.760+0000 7f3536c8e640 1 -- 172.21.15.107:0/1568656730 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3530067ec0 con 0x7f353006a810 2023-12-04T05:42:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.760+0000 7f35257fa640 1 -- 172.21.15.107:0/1568656730 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f352c014960 con 0x7f353006a810 2023-12-04T05:42:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.760+0000 7f35257fa640 1 --2- 172.21.15.107:0/1568656730 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f351c066900 0x7f351c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:50.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.760+0000 7f35257fa640 1 -- 172.21.15.107:0/1568656730 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f352c0850a0 con 0x7f353006a810 2023-12-04T05:42:50.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.760+0000 7f3527fff640 1 --2- 172.21.15.107:0/1568656730 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f351c066900 0x7f351c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:50.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.760+0000 7f3527fff640 1 --2- 172.21.15.107:0/1568656730 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f351c066900 0x7f351c068dc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f35200ae000 tx=0x7f3520009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:50.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.764+0000 7f35257fa640 1 -- 172.21.15.107:0/1568656730 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f352c052210 con 0x7f353006a810 2023-12-04T05:42:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.956+0000 7f3536c8e640 1 -- 172.21.15.107:0/1568656730 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f353006f130 con 0x7f353006a810 2023-12-04T05:42:50.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.956+0000 7f35257fa640 1 -- 172.21.15.107:0/1568656730 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5772 (secure 0 0 0) 0x7f352c055830 con 0x7f353006a810 2023-12-04T05:42:50.962 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:42:50.963 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:42:40.730334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:active","state_seq":45,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:42:50.963 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 22 2023-12-04T05:42:50.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.960+0000 7f3536c8e640 1 -- 172.21.15.107:0/1568656730 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f351c066900 msgr2=0x7f351c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:50.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.960+0000 7f3536c8e640 1 --2- 172.21.15.107:0/1568656730 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f351c066900 0x7f351c068dc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f35200ae000 tx=0x7f3520009040 comp rx=0 tx=0).stop 2023-12-04T05:42:50.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.960+0000 7f3536c8e640 1 -- 172.21.15.107:0/1568656730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f353006a810 msgr2=0x7f35301f62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.960+0000 7f3536c8e640 1 --2- 172.21.15.107:0/1568656730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f353006a810 0x7f35301f62f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f352c00efc0 tx=0x7f352c00c490 comp rx=0 tx=0).stop 2023-12-04T05:42:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.960+0000 7f3536c8e640 1 -- 172.21.15.107:0/1568656730 shutdown_connections 2023-12-04T05:42:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.960+0000 7f3536c8e640 1 --2- 172.21.15.107:0/1568656730 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f351c066900 0x7f351c068dc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.960+0000 7f3536c8e640 1 --2- 172.21.15.107:0/1568656730 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f353014ad30 0x7f353014bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.960+0000 7f3536c8e640 1 --2- 172.21.15.107:0/1568656730 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f353006abf0 0x7f353014f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.960+0000 7f3536c8e640 1 --2- 172.21.15.107:0/1568656730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f353006a810 0x7f35301f62f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.960+0000 7f3536c8e640 1 -- 172.21.15.107:0/1568656730 >> 172.21.15.107:0/1568656730 conn(0x7f35300cc440 msgr2=0x7f35300c7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.960+0000 7f3536c8e640 1 -- 172.21.15.107:0/1568656730 shutdown_connections 2023-12-04T05:42:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:50.960+0000 7f3536c8e640 1 -- 172.21.15.107:0/1568656730 wait complete. 2023-12-04T05:42:50.981 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2023-12-04T05:42:51.912 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2023-12-04T05:42:51.941 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2023-12-04T05:42:52.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:42:53.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.104+0000 7f000d90b640 1 Processor -- start 2023-12-04T05:42:53.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.108+0000 7f000d90b640 1 -- start start 2023-12-04T05:42:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.108+0000 7f000d90b640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f000815d700 0x7f000815dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.108+0000 7f000d90b640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f00080c6380 0x7f000806ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.108+0000 7f000d90b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0008072e50 0x7f000806b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.108+0000 7f000d90b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f000815e020 con 0x7f0008072e50 2023-12-04T05:42:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.108+0000 7f000d90b640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0008071130 con 0x7f000815d700 2023-12-04T05:42:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.108+0000 7f000d90b640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0008071410 con 0x7f00080c6380 2023-12-04T05:42:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.108+0000 7f00077fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0008072e50 0x7f000806b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:53.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.108+0000 7f00077fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0008072e50 0x7f000806b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58166/0 (socket says 172.21.15.107:58166) 2023-12-04T05:42:53.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.108+0000 7f00077fe640 1 -- 172.21.15.107:0/240871230 learned_addr learned my addr 172.21.15.107:0/240871230 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:53.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.108+0000 7f0006ffd640 1 --2- 172.21.15.107:0/240871230 >> v2:172.21.15.195:3300/0 conn(0x7f000815d700 0x7f000815dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:53.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.108+0000 7efffffff640 1 --2- 172.21.15.107:0/240871230 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f00080c6380 0x7f000806ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:53.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.108+0000 7f00077fe640 1 -- 172.21.15.107:0/240871230 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f00080c6380 msgr2=0x7f000806ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:53.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.108+0000 7f00077fe640 1 --2- 172.21.15.107:0/240871230 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f00080c6380 0x7f000806ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:53.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.108+0000 7f00077fe640 1 -- 172.21.15.107:0/240871230 >> v2:172.21.15.195:3300/0 conn(0x7f000815d700 msgr2=0x7f000815dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:42:53.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.108+0000 7f00077fe640 1 --2- 172.21.15.107:0/240871230 >> v2:172.21.15.195:3300/0 conn(0x7f000815d700 0x7f000815dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:53.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.108+0000 7f00077fe640 1 -- 172.21.15.107:0/240871230 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00080716f0 con 0x7f0008072e50 2023-12-04T05:42:53.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.108+0000 7f0006ffd640 1 --2- 172.21.15.107:0/240871230 >> v2:172.21.15.195:3300/0 conn(0x7f000815d700 0x7f000815dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:42:53.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.108+0000 7f00077fe640 1 --2- 172.21.15.107:0/240871230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0008072e50 0x7f000806b1d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f00080663c0 tx=0x7f000001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb45cd581aa2b1 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:53.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.108+0000 7f00067fc640 1 -- 172.21.15.107:0/240871230 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00000af400 con 0x7f0008072e50 2023-12-04T05:42:53.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.108+0000 7f00067fc640 1 -- 172.21.15.107:0/240871230 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00000ab050 con 0x7f0008072e50 2023-12-04T05:42:53.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.108+0000 7f00067fc640 1 -- 172.21.15.107:0/240871230 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00000af780 con 0x7f0008072e50 2023-12-04T05:42:53.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f000d90b640 1 -- 172.21.15.107:0/240871230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0008072e50 msgr2=0x7f000806b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:53.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f000d90b640 1 --2- 172.21.15.107:0/240871230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0008072e50 0x7f000806b1d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f00080663c0 tx=0x7f000001f1c0 comp rx=0 tx=0).stop 2023-12-04T05:42:53.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f000d90b640 1 -- 172.21.15.107:0/240871230 shutdown_connections 2023-12-04T05:42:53.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f000d90b640 1 --2- 172.21.15.107:0/240871230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0008072e50 0x7f000806b1d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:53.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f000d90b640 1 --2- 172.21.15.107:0/240871230 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f00080c6380 0x7f000806ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:53.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f000d90b640 1 --2- 172.21.15.107:0/240871230 >> v2:172.21.15.195:3300/0 conn(0x7f000815d700 0x7f000815dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:53.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f000d90b640 1 -- 172.21.15.107:0/240871230 >> 172.21.15.107:0/240871230 conn(0x7f000806e6f0 msgr2=0x7f00080c8580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:53.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f000d90b640 1 -- 172.21.15.107:0/240871230 shutdown_connections 2023-12-04T05:42:53.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f000d90b640 1 -- 172.21.15.107:0/240871230 wait complete. 2023-12-04T05:42:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f000d90b640 1 Processor -- start 2023-12-04T05:42:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f000d90b640 1 -- start start 2023-12-04T05:42:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f000d90b640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0008072e50 0x7f0008150890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f000d90b640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00080c6380 0x7f00081f02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f000d90b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f000815d700 0x7f000815c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f000d90b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00081efea0 con 0x7f000815d700 2023-12-04T05:42:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f000d90b640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00081587b0 con 0x7f00080c6380 2023-12-04T05:42:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f000d90b640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0008158a50 con 0x7f0008072e50 2023-12-04T05:42:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f00077fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f000815d700 0x7f000815c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f00077fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f000815d700 0x7f000815c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58170/0 (socket says 172.21.15.107:58170) 2023-12-04T05:42:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f00077fe640 1 -- 172.21.15.107:0/3283061772 learned_addr learned my addr 172.21.15.107:0/3283061772 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:42:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f0006ffd640 1 --2- 172.21.15.107:0/3283061772 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0008072e50 0x7f0008150890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7efffffff640 1 --2- 172.21.15.107:0/3283061772 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00080c6380 0x7f00081f02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f00077fe640 1 -- 172.21.15.107:0/3283061772 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0008072e50 msgr2=0x7f0008150890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f00077fe640 1 --2- 172.21.15.107:0/3283061772 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0008072e50 0x7f0008150890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f00077fe640 1 -- 172.21.15.107:0/3283061772 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00080c6380 msgr2=0x7f00081f02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f00077fe640 1 --2- 172.21.15.107:0/3283061772 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00080c6380 0x7f00081f02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f00077fe640 1 -- 172.21.15.107:0/3283061772 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0000009b30 con 0x7f000815d700 2023-12-04T05:42:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f0006ffd640 1 --2- 172.21.15.107:0/3283061772 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0008072e50 0x7f0008150890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:42:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f00077fe640 1 --2- 172.21.15.107:0/3283061772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f000815d700 0x7f000815c550 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f00000b2040 tx=0x7f000001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f0004ff9640 1 -- 172.21.15.107:0/3283061772 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00000afdc0 con 0x7f000815d700 2023-12-04T05:42:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f0004ff9640 1 -- 172.21.15.107:0/3283061772 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0000009510 con 0x7f000815d700 2023-12-04T05:42:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f000d90b640 1 -- 172.21.15.107:0/3283061772 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0008150310 con 0x7f000815d700 2023-12-04T05:42:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f0004ff9640 1 -- 172.21.15.107:0/3283061772 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00000ba660 con 0x7f000815d700 2023-12-04T05:42:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.112+0000 7f000d90b640 1 -- 172.21.15.107:0/3283061772 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00081505b0 con 0x7f000815d700 2023-12-04T05:42:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.116+0000 7f000d90b640 1 -- 172.21.15.107:0/3283061772 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00081effe0 con 0x7f000815d700 2023-12-04T05:42:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.116+0000 7f0004ff9640 1 -- 172.21.15.107:0/3283061772 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f00000b6040 con 0x7f000815d700 2023-12-04T05:42:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.116+0000 7f0004ff9640 1 --2- 172.21.15.107:0/3283061772 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7effe80668d0 0x7effe8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:42:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.116+0000 7f0004ff9640 1 -- 172.21.15.107:0/3283061772 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f0000122930 con 0x7f000815d700 2023-12-04T05:42:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.116+0000 7f0006ffd640 1 --2- 172.21.15.107:0/3283061772 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7effe80668d0 0x7effe8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:42:53.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.116+0000 7f0006ffd640 1 --2- 172.21.15.107:0/3283061772 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7effe80668d0 0x7effe8068d90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7efff8008890 tx=0x7efff8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:42:53.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.120+0000 7f0004ff9640 1 -- 172.21.15.107:0/3283061772 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f00081effe0 con 0x7f000815d700 2023-12-04T05:42:53.263 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2023-12-04T05:42:53.285 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2023-12-04T05:42:53.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.308+0000 7f000d90b640 1 -- 172.21.15.107:0/3283061772 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00081effe0 con 0x7f000815d700 2023-12-04T05:42:53.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.308+0000 7f0004ff9640 1 -- 172.21.15.107:0/3283061772 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6565 (secure 0 0 0) 0x7f00081effe0 con 0x7f000815d700 2023-12-04T05:42:53.315 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:42:53.315 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4747,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6836/2222928136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2222928136},{"type":"v1","addr":"172.21.15.195:6836","nonce":2222928136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:42:40.730334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:active","state_seq":45,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.195:6833/569093002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":569093002},{"type":"v1","addr":"172.21.15.195:6833","nonce":569093002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:42:53.315 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 23 2023-12-04T05:42:53.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.312+0000 7f000d90b640 1 -- 172.21.15.107:0/3283061772 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7effe80668d0 msgr2=0x7effe8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:53.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.312+0000 7f000d90b640 1 --2- 172.21.15.107:0/3283061772 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7effe80668d0 0x7effe8068d90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7efff8008890 tx=0x7efff8008040 comp rx=0 tx=0).stop 2023-12-04T05:42:53.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.312+0000 7f000d90b640 1 -- 172.21.15.107:0/3283061772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f000815d700 msgr2=0x7f000815c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:42:53.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.312+0000 7f000d90b640 1 --2- 172.21.15.107:0/3283061772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f000815d700 0x7f000815c550 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f00000b2040 tx=0x7f000001bf60 comp rx=0 tx=0).stop 2023-12-04T05:42:53.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.312+0000 7f000d90b640 1 -- 172.21.15.107:0/3283061772 shutdown_connections 2023-12-04T05:42:53.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.312+0000 7f000d90b640 1 --2- 172.21.15.107:0/3283061772 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7effe80668d0 0x7effe8068d90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:53.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.312+0000 7f000d90b640 1 --2- 172.21.15.107:0/3283061772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f000815d700 0x7f000815c550 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:53.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.312+0000 7f000d90b640 1 --2- 172.21.15.107:0/3283061772 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f00080c6380 0x7f00081f02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:53.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.312+0000 7f000d90b640 1 --2- 172.21.15.107:0/3283061772 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0008072e50 0x7f0008150890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:42:53.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.312+0000 7f000d90b640 1 -- 172.21.15.107:0/3283061772 >> 172.21.15.107:0/3283061772 conn(0x7f000806e6f0 msgr2=0x7f00080c85f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:42:53.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.312+0000 7f000d90b640 1 -- 172.21.15.107:0/3283061772 shutdown_connections 2023-12-04T05:42:53.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:42:53.312+0000 7f000d90b640 1 -- 172.21.15.107:0/3283061772 wait complete. 2023-12-04T05:42:53.334 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2023-12-04T05:42:53.334 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 108 secs before thrashing 2023-12-04T05:42:54.534 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2023-12-04T05:42:55.460 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2023-12-04T05:42:58.063 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2023-12-04T05:42:58.111 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2023-12-04T05:42:58.420 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2023-12-04T05:42:58.480 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2023-12-04T05:42:58.986 INFO:tasks.workunit.client.0.smithi107.stderr:ffsb_op.c: In function 'print_op_results': 2023-12-04T05:42:58.986 INFO:tasks.workunit.client.0.smithi107.stderr:ffsb_op.c:83:37: warning: embedded '\0' in format [-Wformat-contains-nul] 2023-12-04T05:42:58.986 INFO:tasks.workunit.client.0.smithi107.stderr: 83 | sprintf(buf, "%s/sec\0", buf); 2023-12-04T05:42:58.986 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~ 2023-12-04T05:42:58.986 INFO:tasks.workunit.client.0.smithi107.stderr:ffsb_op.c:86:33: warning: embedded '\0' in format [-Wformat-contains-nul] 2023-12-04T05:42:58.986 INFO:tasks.workunit.client.0.smithi107.stderr: 86 | sprintf(buf, "NA\0"); 2023-12-04T05:42:58.986 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~ 2023-12-04T05:42:59.067 INFO:tasks.workunit.client.0.smithi107.stderr:ffsb_op.c: In function 'print_results': 2023-12-04T05:42:59.067 INFO:tasks.workunit.client.0.smithi107.stderr:ffsb_op.c:83:33: warning: '/sec' directive writing 4 bytes into a region of size between 1 and 256 [-Wformat-overflow=] 2023-12-04T05:42:59.067 INFO:tasks.workunit.client.0.smithi107.stderr: 83 | sprintf(buf, "%s/sec\0", buf); 2023-12-04T05:42:59.067 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~ 2023-12-04T05:42:59.067 INFO:tasks.workunit.client.0.smithi107.stderr:In file included from /usr/include/stdio.h:894, 2023-12-04T05:42:59.067 INFO:tasks.workunit.client.0.smithi107.stderr: from ffsb_op.c:20: 2023-12-04T05:42:59.067 INFO:tasks.workunit.client.0.smithi107.stderr:/usr/include/x86_64-linux-gnu/bits/stdio2.h:38:10: note: '__builtin___sprintf_chk' output between 5 and 260 bytes into a destination of size 256 2023-12-04T05:42:59.067 INFO:tasks.workunit.client.0.smithi107.stderr: 38 | return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, 2023-12-04T05:42:59.067 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-04T05:42:59.068 INFO:tasks.workunit.client.0.smithi107.stderr: 39 | __glibc_objsize (__s), __fmt, 2023-12-04T05:42:59.068 INFO:tasks.workunit.client.0.smithi107.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-04T05:42:59.068 INFO:tasks.workunit.client.0.smithi107.stderr: 40 | __va_arg_pack ()); 2023-12-04T05:42:59.068 INFO:tasks.workunit.client.0.smithi107.stderr: | ~~~~~~~~~~~~~~~~~ 2023-12-04T05:42:59.218 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2023-12-04T05:42:59.270 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2023-12-04T05:43:00.306 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:43:00.308 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:43:00.389 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/util.Tpo .deps/util.Po 2023-12-04T05:43:00.425 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2023-12-04T05:43:00.582 INFO:tasks.workunit.client.0.smithi107.stderr:In file included from parser.h:22, 2023-12-04T05:43:00.582 INFO:tasks.workunit.client.0.smithi107.stderr: from parser.c:25: 2023-12-04T05:43:00.583 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c: In function 'init_filesys': 2023-12-04T05:43:00.584 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-12-04T05:43:00.584 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-12-04T05:43:00.584 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2023-12-04T05:43:00.584 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-12-04T05:43:00.585 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-12-04T05:43:00.585 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2023-12-04T05:43:00.585 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-12-04T05:43:00.585 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2023-12-04T05:43:00.585 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2023-12-04T05:43:00.585 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2023-12-04T05:43:00.585 INFO:tasks.workunit.client.0.smithi107.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2023-12-04T05:43:00.585 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2023-12-04T05:43:00.585 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:934:17: note: in expansion of macro 'list_for_each_entry' 2023-12-04T05:43:00.585 INFO:tasks.workunit.client.0.smithi107.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2023-12-04T05:43:00.585 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-12-04T05:43:00.585 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-12-04T05:43:00.585 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-12-04T05:43:00.585 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2023-12-04T05:43:00.585 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-12-04T05:43:00.586 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-12-04T05:43:00.586 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2023-12-04T05:43:00.586 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-12-04T05:43:00.586 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2023-12-04T05:43:00.586 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2023-12-04T05:43:00.586 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2023-12-04T05:43:00.586 INFO:tasks.workunit.client.0.smithi107.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2023-12-04T05:43:00.586 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2023-12-04T05:43:00.586 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:934:17: note: in expansion of macro 'list_for_each_entry' 2023-12-04T05:43:00.586 INFO:tasks.workunit.client.0.smithi107.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2023-12-04T05:43:00.586 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-12-04T05:43:00.586 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-12-04T05:43:00.586 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-12-04T05:43:00.586 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2023-12-04T05:43:00.587 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-12-04T05:43:00.587 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-12-04T05:43:00.587 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2023-12-04T05:43:00.587 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-12-04T05:43:00.587 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2023-12-04T05:43:00.587 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2023-12-04T05:43:00.587 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2023-12-04T05:43:00.587 INFO:tasks.workunit.client.0.smithi107.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2023-12-04T05:43:00.587 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2023-12-04T05:43:00.587 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:941:17: note: in expansion of macro 'list_for_each_entry' 2023-12-04T05:43:00.587 INFO:tasks.workunit.client.0.smithi107.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2023-12-04T05:43:00.587 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-12-04T05:43:00.587 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-12-04T05:43:00.587 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-12-04T05:43:00.587 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2023-12-04T05:43:00.587 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-12-04T05:43:00.588 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-12-04T05:43:00.588 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2023-12-04T05:43:00.588 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-12-04T05:43:00.588 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2023-12-04T05:43:00.588 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2023-12-04T05:43:00.588 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2023-12-04T05:43:00.588 INFO:tasks.workunit.client.0.smithi107.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2023-12-04T05:43:00.588 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2023-12-04T05:43:00.588 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:941:17: note: in expansion of macro 'list_for_each_entry' 2023-12-04T05:43:00.588 INFO:tasks.workunit.client.0.smithi107.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2023-12-04T05:43:00.588 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-12-04T05:43:00.588 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c: In function 'init_tg_stats': 2023-12-04T05:43:00.588 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-12-04T05:43:00.588 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-12-04T05:43:00.588 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2023-12-04T05:43:00.589 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-12-04T05:43:00.589 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-12-04T05:43:00.589 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2023-12-04T05:43:00.589 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-12-04T05:43:00.589 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2023-12-04T05:43:00.589 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2023-12-04T05:43:00.589 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2023-12-04T05:43:00.589 INFO:tasks.workunit.client.0.smithi107.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2023-12-04T05:43:00.589 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2023-12-04T05:43:00.589 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:971:41: note: in expansion of macro 'list_for_each_entry' 2023-12-04T05:43:00.589 INFO:tasks.workunit.client.0.smithi107.stderr: 971 | list_for_each_entry(tmp_list, 2023-12-04T05:43:00.589 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-12-04T05:43:00.589 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-12-04T05:43:00.589 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-12-04T05:43:00.589 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2023-12-04T05:43:00.590 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-12-04T05:43:00.590 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-12-04T05:43:00.590 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2023-12-04T05:43:00.590 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-12-04T05:43:00.590 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2023-12-04T05:43:00.590 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2023-12-04T05:43:00.590 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2023-12-04T05:43:00.590 INFO:tasks.workunit.client.0.smithi107.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2023-12-04T05:43:00.590 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2023-12-04T05:43:00.590 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:971:41: note: in expansion of macro 'list_for_each_entry' 2023-12-04T05:43:00.590 INFO:tasks.workunit.client.0.smithi107.stderr: 971 | list_for_each_entry(tmp_list, 2023-12-04T05:43:00.590 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-12-04T05:43:00.590 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-12-04T05:43:00.590 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-12-04T05:43:00.590 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2023-12-04T05:43:00.591 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-12-04T05:43:00.591 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-12-04T05:43:00.591 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2023-12-04T05:43:00.591 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-12-04T05:43:00.591 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2023-12-04T05:43:00.591 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2023-12-04T05:43:00.591 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2023-12-04T05:43:00.591 INFO:tasks.workunit.client.0.smithi107.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2023-12-04T05:43:00.591 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2023-12-04T05:43:00.591 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:980:41: note: in expansion of macro 'list_for_each_entry' 2023-12-04T05:43:00.591 INFO:tasks.workunit.client.0.smithi107.stderr: 980 | list_for_each_entry(tmp_list, 2023-12-04T05:43:00.591 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-12-04T05:43:00.591 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-12-04T05:43:00.591 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-12-04T05:43:00.591 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2023-12-04T05:43:00.592 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-12-04T05:43:00.592 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-12-04T05:43:00.592 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2023-12-04T05:43:00.592 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-12-04T05:43:00.592 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2023-12-04T05:43:00.592 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2023-12-04T05:43:00.592 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2023-12-04T05:43:00.592 INFO:tasks.workunit.client.0.smithi107.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2023-12-04T05:43:00.592 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2023-12-04T05:43:00.592 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:980:41: note: in expansion of macro 'list_for_each_entry' 2023-12-04T05:43:00.592 INFO:tasks.workunit.client.0.smithi107.stderr: 980 | list_for_each_entry(tmp_list, 2023-12-04T05:43:00.592 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-12-04T05:43:00.637 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c: In function 'get_optstr': 2023-12-04T05:43:00.637 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:206:23: warning: 'strnlen' specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2023-12-04T05:43:00.637 INFO:tasks.workunit.client.0.smithi107.stderr: 206 | len = strnlen(temp, 4095) + 1; 2023-12-04T05:43:00.637 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-12-04T05:43:00.637 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:200:14: note: source object declared here 2023-12-04T05:43:00.637 INFO:tasks.workunit.client.0.smithi107.stderr: 200 | char temp[BUFSIZE]; 2023-12-04T05:43:00.637 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~ 2023-12-04T05:43:00.654 INFO:tasks.workunit.client.0.smithi107.stderr:In file included from /usr/include/string.h:535, 2023-12-04T05:43:00.654 INFO:tasks.workunit.client.0.smithi107.stderr: from parser.c:19: 2023-12-04T05:43:00.654 INFO:tasks.workunit.client.0.smithi107.stderr:In function 'memcpy', 2023-12-04T05:43:00.654 INFO:tasks.workunit.client.0.smithi107.stderr: inlined from 'size64_convert' at parser.c:119:2: 2023-12-04T05:43:00.654 INFO:tasks.workunit.client.0.smithi107.stderr:/usr/include/x86_64-linux-gnu/bits/string_fortified.h:29:10: warning: '__builtin_memcpy' reading 3 bytes from a region of size 2 [-Wstringop-overread] 2023-12-04T05:43:00.654 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | return __builtin___memcpy_chk (__dest, __src, __len, 2023-12-04T05:43:00.654 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-04T05:43:00.654 INFO:tasks.workunit.client.0.smithi107.stderr: 30 | __glibc_objsize0 (__dest)); 2023-12-04T05:43:00.654 INFO:tasks.workunit.client.0.smithi107.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-04T05:43:00.995 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2023-12-04T05:43:01.030 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2023-12-04T05:43:01.298 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2023-12-04T05:43:01.342 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2023-12-04T05:43:02.570 INFO:tasks.ceph.mds.d.smithi195.stderr:./src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7ff3bf462640 time 2023-12-04T05:43:02.570915+0000 2023-12-04T05:43:02.570 INFO:tasks.ceph.mds.d.smithi195.stderr:./src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-12-04T05:43:02.572 INFO:tasks.ceph.mds.d.smithi195.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:43:02.572 INFO:tasks.ceph.mds.d.smithi195.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7ff3c558fe96] 2023-12-04T05:43:02.572 INFO:tasks.ceph.mds.d.smithi195.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7ff3c559004a] 2023-12-04T05:43:02.572 INFO:tasks.ceph.mds.d.smithi195.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x564b14b293e8] 2023-12-04T05:43:02.572 INFO:tasks.ceph.mds.d.smithi195.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x564b14b1e174] 2023-12-04T05:43:02.572 INFO:tasks.ceph.mds.d.smithi195.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x564b14acf033] 2023-12-04T05:43:02.572 INFO:tasks.ceph.mds.d.smithi195.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x564b14a09dec] 2023-12-04T05:43:02.572 INFO:tasks.ceph.mds.d.smithi195.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x564b14a10be8] 2023-12-04T05:43:02.572 INFO:tasks.ceph.mds.d.smithi195.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x564b14a06a91] 2023-12-04T05:43:02.572 INFO:tasks.ceph.mds.d.smithi195.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x564b14a0734c] 2023-12-04T05:43:02.573 INFO:tasks.ceph.mds.d.smithi195.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564b149f3837] 2023-12-04T05:43:02.573 INFO:tasks.ceph.mds.d.smithi195.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x564b149775fb] 2023-12-04T05:43:02.573 INFO:tasks.ceph.mds.d.smithi195.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564b14977acc] 2023-12-04T05:43:02.573 INFO:tasks.ceph.mds.d.smithi195.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x564b149628e5] 2023-12-04T05:43:02.573 INFO:tasks.ceph.mds.d.smithi195.stderr: 14: (DispatchQueue::entry()+0x5ea) [0x7ff3c577c11a] 2023-12-04T05:43:02.573 INFO:tasks.ceph.mds.d.smithi195.stderr: 15: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7ff3c580e991] 2023-12-04T05:43:02.573 INFO:tasks.ceph.mds.d.smithi195.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff3c4f1cb43] 2023-12-04T05:43:02.573 INFO:tasks.ceph.mds.d.smithi195.stderr: 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff3c4faea00] 2023-12-04T05:43:02.573 INFO:tasks.ceph.mds.d.smithi195.stderr:*** Caught signal (Aborted) ** 2023-12-04T05:43:02.573 INFO:tasks.ceph.mds.d.smithi195.stderr: in thread 7ff3bf462640 thread_name:ms_dispatch 2023-12-04T05:43:02.573 INFO:tasks.ceph.mds.d.smithi195.stderr:2023-12-04T05:43:02.569+0000 7ff3bf462640 -1 ./src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7ff3bf462640 time 2023-12-04T05:43:02.570915+0000 2023-12-04T05:43:02.573 INFO:tasks.ceph.mds.d.smithi195.stderr:./src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-12-04T05:43:02.573 INFO:tasks.ceph.mds.d.smithi195.stderr: 2023-12-04T05:43:02.573 INFO:tasks.ceph.mds.d.smithi195.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:43:02.574 INFO:tasks.ceph.mds.d.smithi195.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7ff3c558fe96] 2023-12-04T05:43:02.574 INFO:tasks.ceph.mds.d.smithi195.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7ff3c559004a] 2023-12-04T05:43:02.574 INFO:tasks.ceph.mds.d.smithi195.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x564b14b293e8] 2023-12-04T05:43:02.574 INFO:tasks.ceph.mds.d.smithi195.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x564b14b1e174] 2023-12-04T05:43:02.574 INFO:tasks.ceph.mds.d.smithi195.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x564b14acf033] 2023-12-04T05:43:02.574 INFO:tasks.ceph.mds.d.smithi195.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x564b14a09dec] 2023-12-04T05:43:02.574 INFO:tasks.ceph.mds.d.smithi195.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x564b14a10be8] 2023-12-04T05:43:02.574 INFO:tasks.ceph.mds.d.smithi195.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x564b14a06a91] 2023-12-04T05:43:02.574 INFO:tasks.ceph.mds.d.smithi195.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x564b14a0734c] 2023-12-04T05:43:02.574 INFO:tasks.ceph.mds.d.smithi195.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564b149f3837] 2023-12-04T05:43:02.574 INFO:tasks.ceph.mds.d.smithi195.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x564b149775fb] 2023-12-04T05:43:02.574 INFO:tasks.ceph.mds.d.smithi195.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564b14977acc] 2023-12-04T05:43:02.574 INFO:tasks.ceph.mds.d.smithi195.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x564b149628e5] 2023-12-04T05:43:02.574 INFO:tasks.ceph.mds.d.smithi195.stderr: 14: (DispatchQueue::entry()+0x5ea) [0x7ff3c577c11a] 2023-12-04T05:43:02.574 INFO:tasks.ceph.mds.d.smithi195.stderr: 15: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7ff3c580e991] 2023-12-04T05:43:02.575 INFO:tasks.ceph.mds.d.smithi195.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff3c4f1cb43] 2023-12-04T05:43:02.575 INFO:tasks.ceph.mds.d.smithi195.stderr: 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff3c4faea00] 2023-12-04T05:43:02.575 INFO:tasks.ceph.mds.d.smithi195.stderr: 2023-12-04T05:43:02.575 INFO:tasks.ceph.mds.d.smithi195.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:43:02.575 INFO:tasks.ceph.mds.d.smithi195.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff3c4eca520] 2023-12-04T05:43:02.575 INFO:tasks.ceph.mds.d.smithi195.stderr: 2: pthread_kill() 2023-12-04T05:43:02.575 INFO:tasks.ceph.mds.d.smithi195.stderr: 3: raise() 2023-12-04T05:43:02.575 INFO:tasks.ceph.mds.d.smithi195.stderr: 4: abort() 2023-12-04T05:43:02.576 INFO:tasks.ceph.mds.d.smithi195.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7ff3c558fef0] 2023-12-04T05:43:02.576 INFO:tasks.ceph.mds.d.smithi195.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7ff3c559004a] 2023-12-04T05:43:02.576 INFO:tasks.ceph.mds.d.smithi195.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x564b14b293e8] 2023-12-04T05:43:02.576 INFO:tasks.ceph.mds.d.smithi195.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x564b14b1e174] 2023-12-04T05:43:02.576 INFO:tasks.ceph.mds.d.smithi195.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x564b14acf033] 2023-12-04T05:43:02.576 INFO:tasks.ceph.mds.d.smithi195.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x564b14a09dec] 2023-12-04T05:43:02.576 INFO:tasks.ceph.mds.d.smithi195.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x564b14a10be8] 2023-12-04T05:43:02.576 INFO:tasks.ceph.mds.d.smithi195.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x564b14a06a91] 2023-12-04T05:43:02.576 INFO:tasks.ceph.mds.d.smithi195.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x564b14a0734c] 2023-12-04T05:43:02.576 INFO:tasks.ceph.mds.d.smithi195.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564b149f3837] 2023-12-04T05:43:02.576 INFO:tasks.ceph.mds.d.smithi195.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x564b149775fb] 2023-12-04T05:43:02.576 INFO:tasks.ceph.mds.d.smithi195.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564b14977acc] 2023-12-04T05:43:02.576 INFO:tasks.ceph.mds.d.smithi195.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x564b149628e5] 2023-12-04T05:43:02.576 INFO:tasks.ceph.mds.d.smithi195.stderr: 18: (DispatchQueue::entry()+0x5ea) [0x7ff3c577c11a] 2023-12-04T05:43:02.576 INFO:tasks.ceph.mds.d.smithi195.stderr: 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7ff3c580e991] 2023-12-04T05:43:02.577 INFO:tasks.ceph.mds.d.smithi195.stderr: 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff3c4f1cb43] 2023-12-04T05:43:02.577 INFO:tasks.ceph.mds.d.smithi195.stderr: 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff3c4faea00] 2023-12-04T05:43:02.577 INFO:tasks.ceph.mds.d.smithi195.stderr:2023-12-04T05:43:02.573+0000 7ff3bf462640 -1 *** Caught signal (Aborted) ** 2023-12-04T05:43:02.577 INFO:tasks.ceph.mds.d.smithi195.stderr: in thread 7ff3bf462640 thread_name:ms_dispatch 2023-12-04T05:43:02.577 INFO:tasks.ceph.mds.d.smithi195.stderr: 2023-12-04T05:43:02.577 INFO:tasks.ceph.mds.d.smithi195.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:43:02.577 INFO:tasks.ceph.mds.d.smithi195.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff3c4eca520] 2023-12-04T05:43:02.577 INFO:tasks.ceph.mds.d.smithi195.stderr: 2: pthread_kill() 2023-12-04T05:43:02.577 INFO:tasks.ceph.mds.d.smithi195.stderr: 3: raise() 2023-12-04T05:43:02.577 INFO:tasks.ceph.mds.d.smithi195.stderr: 4: abort() 2023-12-04T05:43:02.577 INFO:tasks.ceph.mds.d.smithi195.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7ff3c558fef0] 2023-12-04T05:43:02.577 INFO:tasks.ceph.mds.d.smithi195.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7ff3c559004a] 2023-12-04T05:43:02.577 INFO:tasks.ceph.mds.d.smithi195.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x564b14b293e8] 2023-12-04T05:43:02.577 INFO:tasks.ceph.mds.d.smithi195.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x564b14b1e174] 2023-12-04T05:43:02.578 INFO:tasks.ceph.mds.d.smithi195.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x564b14acf033] 2023-12-04T05:43:02.578 INFO:tasks.ceph.mds.d.smithi195.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x564b14a09dec] 2023-12-04T05:43:02.578 INFO:tasks.ceph.mds.d.smithi195.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x564b14a10be8] 2023-12-04T05:43:02.578 INFO:tasks.ceph.mds.d.smithi195.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x564b14a06a91] 2023-12-04T05:43:02.578 INFO:tasks.ceph.mds.d.smithi195.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x564b14a0734c] 2023-12-04T05:43:02.578 INFO:tasks.ceph.mds.d.smithi195.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564b149f3837] 2023-12-04T05:43:02.578 INFO:tasks.ceph.mds.d.smithi195.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x564b149775fb] 2023-12-04T05:43:02.578 INFO:tasks.ceph.mds.d.smithi195.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564b14977acc] 2023-12-04T05:43:02.578 INFO:tasks.ceph.mds.d.smithi195.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x564b149628e5] 2023-12-04T05:43:02.578 INFO:tasks.ceph.mds.d.smithi195.stderr: 18: (DispatchQueue::entry()+0x5ea) [0x7ff3c577c11a] 2023-12-04T05:43:02.578 INFO:tasks.ceph.mds.d.smithi195.stderr: 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7ff3c580e991] 2023-12-04T05:43:02.578 INFO:tasks.ceph.mds.d.smithi195.stderr: 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff3c4f1cb43] 2023-12-04T05:43:02.578 INFO:tasks.ceph.mds.d.smithi195.stderr: 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff3c4faea00] 2023-12-04T05:43:02.578 INFO:tasks.ceph.mds.d.smithi195.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-04T05:43:02.578 INFO:tasks.ceph.mds.d.smithi195.stderr: 2023-12-04T05:43:02.613 INFO:tasks.ceph.mds.d.smithi195.stderr: -1> 2023-12-04T05:43:02.569+0000 7ff3bf462640 -1 ./src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7ff3bf462640 time 2023-12-04T05:43:02.570915+0000 2023-12-04T05:43:02.613 INFO:tasks.ceph.mds.d.smithi195.stderr:./src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-12-04T05:43:02.613 INFO:tasks.ceph.mds.d.smithi195.stderr: 2023-12-04T05:43:02.613 INFO:tasks.ceph.mds.d.smithi195.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:43:02.613 INFO:tasks.ceph.mds.d.smithi195.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7ff3c558fe96] 2023-12-04T05:43:02.613 INFO:tasks.ceph.mds.d.smithi195.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7ff3c559004a] 2023-12-04T05:43:02.613 INFO:tasks.ceph.mds.d.smithi195.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x564b14b293e8] 2023-12-04T05:43:02.613 INFO:tasks.ceph.mds.d.smithi195.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x564b14b1e174] 2023-12-04T05:43:02.614 INFO:tasks.ceph.mds.d.smithi195.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x564b14acf033] 2023-12-04T05:43:02.614 INFO:tasks.ceph.mds.d.smithi195.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x564b14a09dec] 2023-12-04T05:43:02.614 INFO:tasks.ceph.mds.d.smithi195.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x564b14a10be8] 2023-12-04T05:43:02.614 INFO:tasks.ceph.mds.d.smithi195.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x564b14a06a91] 2023-12-04T05:43:02.614 INFO:tasks.ceph.mds.d.smithi195.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x564b14a0734c] 2023-12-04T05:43:02.614 INFO:tasks.ceph.mds.d.smithi195.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564b149f3837] 2023-12-04T05:43:02.614 INFO:tasks.ceph.mds.d.smithi195.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x564b149775fb] 2023-12-04T05:43:02.614 INFO:tasks.ceph.mds.d.smithi195.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564b14977acc] 2023-12-04T05:43:02.614 INFO:tasks.ceph.mds.d.smithi195.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x564b149628e5] 2023-12-04T05:43:02.614 INFO:tasks.ceph.mds.d.smithi195.stderr: 14: (DispatchQueue::entry()+0x5ea) [0x7ff3c577c11a] 2023-12-04T05:43:02.614 INFO:tasks.ceph.mds.d.smithi195.stderr: 15: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7ff3c580e991] 2023-12-04T05:43:02.614 INFO:tasks.ceph.mds.d.smithi195.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff3c4f1cb43] 2023-12-04T05:43:02.614 INFO:tasks.ceph.mds.d.smithi195.stderr: 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff3c4faea00] 2023-12-04T05:43:02.614 INFO:tasks.ceph.mds.d.smithi195.stderr: 2023-12-04T05:43:02.615 INFO:tasks.ceph.mds.d.smithi195.stderr: 0> 2023-12-04T05:43:02.573+0000 7ff3bf462640 -1 *** Caught signal (Aborted) ** 2023-12-04T05:43:02.615 INFO:tasks.ceph.mds.d.smithi195.stderr: in thread 7ff3bf462640 thread_name:ms_dispatch 2023-12-04T05:43:02.615 INFO:tasks.ceph.mds.d.smithi195.stderr: 2023-12-04T05:43:02.615 INFO:tasks.ceph.mds.d.smithi195.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:43:02.615 INFO:tasks.ceph.mds.d.smithi195.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff3c4eca520] 2023-12-04T05:43:02.615 INFO:tasks.ceph.mds.d.smithi195.stderr: 2: pthread_kill() 2023-12-04T05:43:02.615 INFO:tasks.ceph.mds.d.smithi195.stderr: 3: raise() 2023-12-04T05:43:02.615 INFO:tasks.ceph.mds.d.smithi195.stderr: 4: abort() 2023-12-04T05:43:02.615 INFO:tasks.ceph.mds.d.smithi195.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7ff3c558fef0] 2023-12-04T05:43:02.615 INFO:tasks.ceph.mds.d.smithi195.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7ff3c559004a] 2023-12-04T05:43:02.615 INFO:tasks.ceph.mds.d.smithi195.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x564b14b293e8] 2023-12-04T05:43:02.615 INFO:tasks.ceph.mds.d.smithi195.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x564b14b1e174] 2023-12-04T05:43:02.615 INFO:tasks.ceph.mds.d.smithi195.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x564b14acf033] 2023-12-04T05:43:02.615 INFO:tasks.ceph.mds.d.smithi195.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x564b14a09dec] 2023-12-04T05:43:02.615 INFO:tasks.ceph.mds.d.smithi195.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x564b14a10be8] 2023-12-04T05:43:02.615 INFO:tasks.ceph.mds.d.smithi195.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x564b14a06a91] 2023-12-04T05:43:02.616 INFO:tasks.ceph.mds.d.smithi195.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x564b14a0734c] 2023-12-04T05:43:02.616 INFO:tasks.ceph.mds.d.smithi195.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564b149f3837] 2023-12-04T05:43:02.616 INFO:tasks.ceph.mds.d.smithi195.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x564b149775fb] 2023-12-04T05:43:02.616 INFO:tasks.ceph.mds.d.smithi195.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564b14977acc] 2023-12-04T05:43:02.616 INFO:tasks.ceph.mds.d.smithi195.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x564b149628e5] 2023-12-04T05:43:02.616 INFO:tasks.ceph.mds.d.smithi195.stderr: 18: (DispatchQueue::entry()+0x5ea) [0x7ff3c577c11a] 2023-12-04T05:43:02.616 INFO:tasks.ceph.mds.d.smithi195.stderr: 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7ff3c580e991] 2023-12-04T05:43:02.616 INFO:tasks.ceph.mds.d.smithi195.stderr: 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff3c4f1cb43] 2023-12-04T05:43:02.616 INFO:tasks.ceph.mds.d.smithi195.stderr: 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff3c4faea00] 2023-12-04T05:43:02.616 INFO:tasks.ceph.mds.d.smithi195.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-04T05:43:02.616 INFO:tasks.ceph.mds.d.smithi195.stderr: 2023-12-04T05:43:02.635 INFO:tasks.ceph.mds.d.smithi195.stderr: -1> 2023-12-04T05:43:02.569+0000 7ff3bf462640 -1 ./src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7ff3bf462640 time 2023-12-04T05:43:02.570915+0000 2023-12-04T05:43:02.636 INFO:tasks.ceph.mds.d.smithi195.stderr:./src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-12-04T05:43:02.636 INFO:tasks.ceph.mds.d.smithi195.stderr: 2023-12-04T05:43:02.636 INFO:tasks.ceph.mds.d.smithi195.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:43:02.636 INFO:tasks.ceph.mds.d.smithi195.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7ff3c558fe96] 2023-12-04T05:43:02.636 INFO:tasks.ceph.mds.d.smithi195.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7ff3c559004a] 2023-12-04T05:43:02.636 INFO:tasks.ceph.mds.d.smithi195.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x564b14b293e8] 2023-12-04T05:43:02.636 INFO:tasks.ceph.mds.d.smithi195.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x564b14b1e174] 2023-12-04T05:43:02.636 INFO:tasks.ceph.mds.d.smithi195.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x564b14acf033] 2023-12-04T05:43:02.636 INFO:tasks.ceph.mds.d.smithi195.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x564b14a09dec] 2023-12-04T05:43:02.636 INFO:tasks.ceph.mds.d.smithi195.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x564b14a10be8] 2023-12-04T05:43:02.636 INFO:tasks.ceph.mds.d.smithi195.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x564b14a06a91] 2023-12-04T05:43:02.636 INFO:tasks.ceph.mds.d.smithi195.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x564b14a0734c] 2023-12-04T05:43:02.636 INFO:tasks.ceph.mds.d.smithi195.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564b149f3837] 2023-12-04T05:43:02.636 INFO:tasks.ceph.mds.d.smithi195.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x564b149775fb] 2023-12-04T05:43:02.637 INFO:tasks.ceph.mds.d.smithi195.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564b14977acc] 2023-12-04T05:43:02.637 INFO:tasks.ceph.mds.d.smithi195.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x564b149628e5] 2023-12-04T05:43:02.637 INFO:tasks.ceph.mds.d.smithi195.stderr: 14: (DispatchQueue::entry()+0x5ea) [0x7ff3c577c11a] 2023-12-04T05:43:02.637 INFO:tasks.ceph.mds.d.smithi195.stderr: 15: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7ff3c580e991] 2023-12-04T05:43:02.637 INFO:tasks.ceph.mds.d.smithi195.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff3c4f1cb43] 2023-12-04T05:43:02.637 INFO:tasks.ceph.mds.d.smithi195.stderr: 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff3c4faea00] 2023-12-04T05:43:02.637 INFO:tasks.ceph.mds.d.smithi195.stderr: 2023-12-04T05:43:02.637 INFO:tasks.ceph.mds.d.smithi195.stderr: 0> 2023-12-04T05:43:02.573+0000 7ff3bf462640 -1 *** Caught signal (Aborted) ** 2023-12-04T05:43:02.637 INFO:tasks.ceph.mds.d.smithi195.stderr: in thread 7ff3bf462640 thread_name:ms_dispatch 2023-12-04T05:43:02.637 INFO:tasks.ceph.mds.d.smithi195.stderr: 2023-12-04T05:43:02.637 INFO:tasks.ceph.mds.d.smithi195.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:43:02.637 INFO:tasks.ceph.mds.d.smithi195.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff3c4eca520] 2023-12-04T05:43:02.637 INFO:tasks.ceph.mds.d.smithi195.stderr: 2: pthread_kill() 2023-12-04T05:43:02.637 INFO:tasks.ceph.mds.d.smithi195.stderr: 3: raise() 2023-12-04T05:43:02.637 INFO:tasks.ceph.mds.d.smithi195.stderr: 4: abort() 2023-12-04T05:43:02.637 INFO:tasks.ceph.mds.d.smithi195.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7ff3c558fef0] 2023-12-04T05:43:02.638 INFO:tasks.ceph.mds.d.smithi195.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7ff3c559004a] 2023-12-04T05:43:02.638 INFO:tasks.ceph.mds.d.smithi195.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x564b14b293e8] 2023-12-04T05:43:02.638 INFO:tasks.ceph.mds.d.smithi195.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x564b14b1e174] 2023-12-04T05:43:02.638 INFO:tasks.ceph.mds.d.smithi195.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x564b14acf033] 2023-12-04T05:43:02.638 INFO:tasks.ceph.mds.d.smithi195.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x564b14a09dec] 2023-12-04T05:43:02.638 INFO:tasks.ceph.mds.d.smithi195.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x564b14a10be8] 2023-12-04T05:43:02.638 INFO:tasks.ceph.mds.d.smithi195.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x564b14a06a91] 2023-12-04T05:43:02.638 INFO:tasks.ceph.mds.d.smithi195.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x564b14a0734c] 2023-12-04T05:43:02.638 INFO:tasks.ceph.mds.d.smithi195.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564b149f3837] 2023-12-04T05:43:02.638 INFO:tasks.ceph.mds.d.smithi195.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x564b149775fb] 2023-12-04T05:43:02.638 INFO:tasks.ceph.mds.d.smithi195.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564b14977acc] 2023-12-04T05:43:02.638 INFO:tasks.ceph.mds.d.smithi195.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x564b149628e5] 2023-12-04T05:43:02.638 INFO:tasks.ceph.mds.d.smithi195.stderr: 18: (DispatchQueue::entry()+0x5ea) [0x7ff3c577c11a] 2023-12-04T05:43:02.638 INFO:tasks.ceph.mds.d.smithi195.stderr: 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7ff3c580e991] 2023-12-04T05:43:02.638 INFO:tasks.ceph.mds.d.smithi195.stderr: 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff3c4f1cb43] 2023-12-04T05:43:02.639 INFO:tasks.ceph.mds.d.smithi195.stderr: 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff3c4faea00] 2023-12-04T05:43:02.639 INFO:tasks.ceph.mds.d.smithi195.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-04T05:43:02.639 INFO:tasks.ceph.mds.d.smithi195.stderr: 2023-12-04T05:43:03.087 INFO:tasks.ceph.mds.d.smithi195.stderr:daemon-helper: command crashed with signal 6 2023-12-04T05:43:06.937 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2023-12-04T05:43:13.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~7s 2023-12-04T05:43:20.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~14s 2023-12-04T05:43:27.375 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~20s 2023-12-04T05:43:30.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:43:30.356 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:43:34.186 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~27s 2023-12-04T05:43:40.276 INFO:tasks.ceph.mds.f.smithi195.stderr:2023-12-04T05:43:40.272+0000 7f5bdadb8640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-12-04T05:43:40.997 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~34s 2023-12-04T05:43:41.210 INFO:tasks.ceph.mds.b.smithi195.stderr:./src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7f3441abe640 time 2023-12-04T05:43:41.211692+0000 2023-12-04T05:43:41.211 INFO:tasks.ceph.mds.b.smithi195.stderr:./src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-12-04T05:43:41.212 INFO:tasks.ceph.mds.b.smithi195.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:43:41.212 INFO:tasks.ceph.mds.b.smithi195.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f3447bebe96] 2023-12-04T05:43:41.212 INFO:tasks.ceph.mds.b.smithi195.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f3447bec04a] 2023-12-04T05:43:41.212 INFO:tasks.ceph.mds.b.smithi195.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x560532b413e8] 2023-12-04T05:43:41.213 INFO:tasks.ceph.mds.b.smithi195.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x560532b36174] 2023-12-04T05:43:41.213 INFO:tasks.ceph.mds.b.smithi195.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x560532ae7033] 2023-12-04T05:43:41.213 INFO:tasks.ceph.mds.b.smithi195.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x560532a21dec] 2023-12-04T05:43:41.213 INFO:tasks.ceph.mds.b.smithi195.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x560532a28be8] 2023-12-04T05:43:41.213 INFO:tasks.ceph.mds.b.smithi195.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x560532a1ea91] 2023-12-04T05:43:41.213 INFO:tasks.ceph.mds.b.smithi195.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x560532a1f34c] 2023-12-04T05:43:41.213 INFO:tasks.ceph.mds.b.smithi195.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560532a0b837] 2023-12-04T05:43:41.213 INFO:tasks.ceph.mds.b.smithi195.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x56053298f5fb] 2023-12-04T05:43:41.213 INFO:tasks.ceph.mds.b.smithi195.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56053298facc] 2023-12-04T05:43:41.213 INFO:tasks.ceph.mds.b.smithi195.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x56053297a8e5] 2023-12-04T05:43:41.213 INFO:tasks.ceph.mds.b.smithi195.stderr: 14: (DispatchQueue::entry()+0x5ea) [0x7f3447dd811a] 2023-12-04T05:43:41.213 INFO:tasks.ceph.mds.b.smithi195.stderr: 15: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7f3447e6a991] 2023-12-04T05:43:41.213 INFO:tasks.ceph.mds.b.smithi195.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f3447578b43] 2023-12-04T05:43:41.213 INFO:tasks.ceph.mds.b.smithi195.stderr: 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f344760aa00] 2023-12-04T05:43:41.213 INFO:tasks.ceph.mds.b.smithi195.stderr:*** Caught signal (Aborted) ** 2023-12-04T05:43:41.214 INFO:tasks.ceph.mds.b.smithi195.stderr: in thread 7f3441abe640 thread_name:ms_dispatch 2023-12-04T05:43:41.214 INFO:tasks.ceph.mds.b.smithi195.stderr:2023-12-04T05:43:41.208+0000 7f3441abe640 -1 ./src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7f3441abe640 time 2023-12-04T05:43:41.211692+0000 2023-12-04T05:43:41.214 INFO:tasks.ceph.mds.b.smithi195.stderr:./src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-12-04T05:43:41.214 INFO:tasks.ceph.mds.b.smithi195.stderr: 2023-12-04T05:43:41.214 INFO:tasks.ceph.mds.b.smithi195.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:43:41.214 INFO:tasks.ceph.mds.b.smithi195.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f3447bebe96] 2023-12-04T05:43:41.214 INFO:tasks.ceph.mds.b.smithi195.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f3447bec04a] 2023-12-04T05:43:41.214 INFO:tasks.ceph.mds.b.smithi195.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x560532b413e8] 2023-12-04T05:43:41.214 INFO:tasks.ceph.mds.b.smithi195.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x560532b36174] 2023-12-04T05:43:41.214 INFO:tasks.ceph.mds.b.smithi195.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x560532ae7033] 2023-12-04T05:43:41.214 INFO:tasks.ceph.mds.b.smithi195.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x560532a21dec] 2023-12-04T05:43:41.214 INFO:tasks.ceph.mds.b.smithi195.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x560532a28be8] 2023-12-04T05:43:41.214 INFO:tasks.ceph.mds.b.smithi195.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x560532a1ea91] 2023-12-04T05:43:41.214 INFO:tasks.ceph.mds.b.smithi195.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x560532a1f34c] 2023-12-04T05:43:41.214 INFO:tasks.ceph.mds.b.smithi195.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560532a0b837] 2023-12-04T05:43:41.215 INFO:tasks.ceph.mds.b.smithi195.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x56053298f5fb] 2023-12-04T05:43:41.215 INFO:tasks.ceph.mds.b.smithi195.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56053298facc] 2023-12-04T05:43:41.215 INFO:tasks.ceph.mds.b.smithi195.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x56053297a8e5] 2023-12-04T05:43:41.215 INFO:tasks.ceph.mds.b.smithi195.stderr: 14: (DispatchQueue::entry()+0x5ea) [0x7f3447dd811a] 2023-12-04T05:43:41.215 INFO:tasks.ceph.mds.b.smithi195.stderr: 15: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7f3447e6a991] 2023-12-04T05:43:41.215 INFO:tasks.ceph.mds.b.smithi195.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f3447578b43] 2023-12-04T05:43:41.215 INFO:tasks.ceph.mds.b.smithi195.stderr: 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f344760aa00] 2023-12-04T05:43:41.215 INFO:tasks.ceph.mds.b.smithi195.stderr: 2023-12-04T05:43:41.215 INFO:tasks.ceph.mds.b.smithi195.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:43:41.216 INFO:tasks.ceph.mds.b.smithi195.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f3447526520] 2023-12-04T05:43:41.216 INFO:tasks.ceph.mds.b.smithi195.stderr: 2: pthread_kill() 2023-12-04T05:43:41.216 INFO:tasks.ceph.mds.b.smithi195.stderr: 3: raise() 2023-12-04T05:43:41.216 INFO:tasks.ceph.mds.b.smithi195.stderr: 4: abort() 2023-12-04T05:43:41.216 INFO:tasks.ceph.mds.b.smithi195.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f3447bebef0] 2023-12-04T05:43:41.216 INFO:tasks.ceph.mds.b.smithi195.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f3447bec04a] 2023-12-04T05:43:41.216 INFO:tasks.ceph.mds.b.smithi195.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x560532b413e8] 2023-12-04T05:43:41.216 INFO:tasks.ceph.mds.b.smithi195.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x560532b36174] 2023-12-04T05:43:41.216 INFO:tasks.ceph.mds.b.smithi195.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x560532ae7033] 2023-12-04T05:43:41.216 INFO:tasks.ceph.mds.b.smithi195.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x560532a21dec] 2023-12-04T05:43:41.216 INFO:tasks.ceph.mds.b.smithi195.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x560532a28be8] 2023-12-04T05:43:41.216 INFO:tasks.ceph.mds.b.smithi195.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x560532a1ea91] 2023-12-04T05:43:41.216 INFO:tasks.ceph.mds.b.smithi195.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x560532a1f34c] 2023-12-04T05:43:41.216 INFO:tasks.ceph.mds.b.smithi195.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560532a0b837] 2023-12-04T05:43:41.217 INFO:tasks.ceph.mds.b.smithi195.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x56053298f5fb] 2023-12-04T05:43:41.217 INFO:tasks.ceph.mds.b.smithi195.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56053298facc] 2023-12-04T05:43:41.217 INFO:tasks.ceph.mds.b.smithi195.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x56053297a8e5] 2023-12-04T05:43:41.217 INFO:tasks.ceph.mds.b.smithi195.stderr: 18: (DispatchQueue::entry()+0x5ea) [0x7f3447dd811a] 2023-12-04T05:43:41.217 INFO:tasks.ceph.mds.b.smithi195.stderr: 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7f3447e6a991] 2023-12-04T05:43:41.217 INFO:tasks.ceph.mds.b.smithi195.stderr: 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f3447578b43] 2023-12-04T05:43:41.217 INFO:tasks.ceph.mds.b.smithi195.stderr: 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f344760aa00] 2023-12-04T05:43:41.217 INFO:tasks.ceph.mds.b.smithi195.stderr:2023-12-04T05:43:41.212+0000 7f3441abe640 -1 *** Caught signal (Aborted) ** 2023-12-04T05:43:41.217 INFO:tasks.ceph.mds.b.smithi195.stderr: in thread 7f3441abe640 thread_name:ms_dispatch 2023-12-04T05:43:41.217 INFO:tasks.ceph.mds.b.smithi195.stderr: 2023-12-04T05:43:41.217 INFO:tasks.ceph.mds.b.smithi195.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:43:41.217 INFO:tasks.ceph.mds.b.smithi195.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f3447526520] 2023-12-04T05:43:41.217 INFO:tasks.ceph.mds.b.smithi195.stderr: 2: pthread_kill() 2023-12-04T05:43:41.217 INFO:tasks.ceph.mds.b.smithi195.stderr: 3: raise() 2023-12-04T05:43:41.218 INFO:tasks.ceph.mds.b.smithi195.stderr: 4: abort() 2023-12-04T05:43:41.218 INFO:tasks.ceph.mds.b.smithi195.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f3447bebef0] 2023-12-04T05:43:41.218 INFO:tasks.ceph.mds.b.smithi195.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f3447bec04a] 2023-12-04T05:43:41.218 INFO:tasks.ceph.mds.b.smithi195.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x560532b413e8] 2023-12-04T05:43:41.218 INFO:tasks.ceph.mds.b.smithi195.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x560532b36174] 2023-12-04T05:43:41.218 INFO:tasks.ceph.mds.b.smithi195.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x560532ae7033] 2023-12-04T05:43:41.218 INFO:tasks.ceph.mds.b.smithi195.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x560532a21dec] 2023-12-04T05:43:41.218 INFO:tasks.ceph.mds.b.smithi195.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x560532a28be8] 2023-12-04T05:43:41.218 INFO:tasks.ceph.mds.b.smithi195.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x560532a1ea91] 2023-12-04T05:43:41.218 INFO:tasks.ceph.mds.b.smithi195.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x560532a1f34c] 2023-12-04T05:43:41.218 INFO:tasks.ceph.mds.b.smithi195.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560532a0b837] 2023-12-04T05:43:41.218 INFO:tasks.ceph.mds.b.smithi195.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x56053298f5fb] 2023-12-04T05:43:41.218 INFO:tasks.ceph.mds.b.smithi195.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56053298facc] 2023-12-04T05:43:41.218 INFO:tasks.ceph.mds.b.smithi195.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x56053297a8e5] 2023-12-04T05:43:41.218 INFO:tasks.ceph.mds.b.smithi195.stderr: 18: (DispatchQueue::entry()+0x5ea) [0x7f3447dd811a] 2023-12-04T05:43:41.219 INFO:tasks.ceph.mds.b.smithi195.stderr: 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7f3447e6a991] 2023-12-04T05:43:41.219 INFO:tasks.ceph.mds.b.smithi195.stderr: 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f3447578b43] 2023-12-04T05:43:41.219 INFO:tasks.ceph.mds.b.smithi195.stderr: 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f344760aa00] 2023-12-04T05:43:41.219 INFO:tasks.ceph.mds.b.smithi195.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-04T05:43:41.219 INFO:tasks.ceph.mds.b.smithi195.stderr: 2023-12-04T05:43:41.249 INFO:tasks.ceph.mds.b.smithi195.stderr: -1> 2023-12-04T05:43:41.208+0000 7f3441abe640 -1 ./src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7f3441abe640 time 2023-12-04T05:43:41.211692+0000 2023-12-04T05:43:41.249 INFO:tasks.ceph.mds.b.smithi195.stderr:./src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-12-04T05:43:41.249 INFO:tasks.ceph.mds.b.smithi195.stderr: 2023-12-04T05:43:41.250 INFO:tasks.ceph.mds.b.smithi195.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:43:41.250 INFO:tasks.ceph.mds.b.smithi195.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f3447bebe96] 2023-12-04T05:43:41.250 INFO:tasks.ceph.mds.b.smithi195.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f3447bec04a] 2023-12-04T05:43:41.250 INFO:tasks.ceph.mds.b.smithi195.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x560532b413e8] 2023-12-04T05:43:41.250 INFO:tasks.ceph.mds.b.smithi195.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x560532b36174] 2023-12-04T05:43:41.250 INFO:tasks.ceph.mds.b.smithi195.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x560532ae7033] 2023-12-04T05:43:41.250 INFO:tasks.ceph.mds.b.smithi195.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x560532a21dec] 2023-12-04T05:43:41.250 INFO:tasks.ceph.mds.b.smithi195.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x560532a28be8] 2023-12-04T05:43:41.250 INFO:tasks.ceph.mds.b.smithi195.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x560532a1ea91] 2023-12-04T05:43:41.250 INFO:tasks.ceph.mds.b.smithi195.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x560532a1f34c] 2023-12-04T05:43:41.250 INFO:tasks.ceph.mds.b.smithi195.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560532a0b837] 2023-12-04T05:43:41.250 INFO:tasks.ceph.mds.b.smithi195.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x56053298f5fb] 2023-12-04T05:43:41.250 INFO:tasks.ceph.mds.b.smithi195.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56053298facc] 2023-12-04T05:43:41.250 INFO:tasks.ceph.mds.b.smithi195.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x56053297a8e5] 2023-12-04T05:43:41.251 INFO:tasks.ceph.mds.b.smithi195.stderr: 14: (DispatchQueue::entry()+0x5ea) [0x7f3447dd811a] 2023-12-04T05:43:41.251 INFO:tasks.ceph.mds.b.smithi195.stderr: 15: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7f3447e6a991] 2023-12-04T05:43:41.251 INFO:tasks.ceph.mds.b.smithi195.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f3447578b43] 2023-12-04T05:43:41.251 INFO:tasks.ceph.mds.b.smithi195.stderr: 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f344760aa00] 2023-12-04T05:43:41.251 INFO:tasks.ceph.mds.b.smithi195.stderr: 2023-12-04T05:43:41.251 INFO:tasks.ceph.mds.b.smithi195.stderr: 0> 2023-12-04T05:43:41.212+0000 7f3441abe640 -1 *** Caught signal (Aborted) ** 2023-12-04T05:43:41.251 INFO:tasks.ceph.mds.b.smithi195.stderr: in thread 7f3441abe640 thread_name:ms_dispatch 2023-12-04T05:43:41.251 INFO:tasks.ceph.mds.b.smithi195.stderr: 2023-12-04T05:43:41.251 INFO:tasks.ceph.mds.b.smithi195.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:43:41.251 INFO:tasks.ceph.mds.b.smithi195.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f3447526520] 2023-12-04T05:43:41.251 INFO:tasks.ceph.mds.b.smithi195.stderr: 2: pthread_kill() 2023-12-04T05:43:41.251 INFO:tasks.ceph.mds.b.smithi195.stderr: 3: raise() 2023-12-04T05:43:41.251 INFO:tasks.ceph.mds.b.smithi195.stderr: 4: abort() 2023-12-04T05:43:41.251 INFO:tasks.ceph.mds.b.smithi195.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f3447bebef0] 2023-12-04T05:43:41.251 INFO:tasks.ceph.mds.b.smithi195.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f3447bec04a] 2023-12-04T05:43:41.251 INFO:tasks.ceph.mds.b.smithi195.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x560532b413e8] 2023-12-04T05:43:41.252 INFO:tasks.ceph.mds.b.smithi195.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x560532b36174] 2023-12-04T05:43:41.252 INFO:tasks.ceph.mds.b.smithi195.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x560532ae7033] 2023-12-04T05:43:41.252 INFO:tasks.ceph.mds.b.smithi195.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x560532a21dec] 2023-12-04T05:43:41.252 INFO:tasks.ceph.mds.b.smithi195.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x560532a28be8] 2023-12-04T05:43:41.252 INFO:tasks.ceph.mds.b.smithi195.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x560532a1ea91] 2023-12-04T05:43:41.252 INFO:tasks.ceph.mds.b.smithi195.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x560532a1f34c] 2023-12-04T05:43:41.252 INFO:tasks.ceph.mds.b.smithi195.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560532a0b837] 2023-12-04T05:43:41.252 INFO:tasks.ceph.mds.b.smithi195.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x56053298f5fb] 2023-12-04T05:43:41.252 INFO:tasks.ceph.mds.b.smithi195.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56053298facc] 2023-12-04T05:43:41.252 INFO:tasks.ceph.mds.b.smithi195.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x56053297a8e5] 2023-12-04T05:43:41.252 INFO:tasks.ceph.mds.b.smithi195.stderr: 18: (DispatchQueue::entry()+0x5ea) [0x7f3447dd811a] 2023-12-04T05:43:41.252 INFO:tasks.ceph.mds.b.smithi195.stderr: 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7f3447e6a991] 2023-12-04T05:43:41.252 INFO:tasks.ceph.mds.b.smithi195.stderr: 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f3447578b43] 2023-12-04T05:43:41.252 INFO:tasks.ceph.mds.b.smithi195.stderr: 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f344760aa00] 2023-12-04T05:43:41.252 INFO:tasks.ceph.mds.b.smithi195.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-04T05:43:41.253 INFO:tasks.ceph.mds.b.smithi195.stderr: 2023-12-04T05:43:41.270 INFO:tasks.ceph.mds.b.smithi195.stderr: -1> 2023-12-04T05:43:41.208+0000 7f3441abe640 -1 ./src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7f3441abe640 time 2023-12-04T05:43:41.211692+0000 2023-12-04T05:43:41.270 INFO:tasks.ceph.mds.b.smithi195.stderr:./src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-12-04T05:43:41.270 INFO:tasks.ceph.mds.b.smithi195.stderr: 2023-12-04T05:43:41.270 INFO:tasks.ceph.mds.b.smithi195.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:43:41.270 INFO:tasks.ceph.mds.b.smithi195.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f3447bebe96] 2023-12-04T05:43:41.270 INFO:tasks.ceph.mds.b.smithi195.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f3447bec04a] 2023-12-04T05:43:41.270 INFO:tasks.ceph.mds.b.smithi195.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x560532b413e8] 2023-12-04T05:43:41.270 INFO:tasks.ceph.mds.b.smithi195.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x560532b36174] 2023-12-04T05:43:41.270 INFO:tasks.ceph.mds.b.smithi195.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x560532ae7033] 2023-12-04T05:43:41.270 INFO:tasks.ceph.mds.b.smithi195.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x560532a21dec] 2023-12-04T05:43:41.270 INFO:tasks.ceph.mds.b.smithi195.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x560532a28be8] 2023-12-04T05:43:41.271 INFO:tasks.ceph.mds.b.smithi195.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x560532a1ea91] 2023-12-04T05:43:41.271 INFO:tasks.ceph.mds.b.smithi195.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x560532a1f34c] 2023-12-04T05:43:41.271 INFO:tasks.ceph.mds.b.smithi195.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560532a0b837] 2023-12-04T05:43:41.271 INFO:tasks.ceph.mds.b.smithi195.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x56053298f5fb] 2023-12-04T05:43:41.271 INFO:tasks.ceph.mds.b.smithi195.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56053298facc] 2023-12-04T05:43:41.271 INFO:tasks.ceph.mds.b.smithi195.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x56053297a8e5] 2023-12-04T05:43:41.271 INFO:tasks.ceph.mds.b.smithi195.stderr: 14: (DispatchQueue::entry()+0x5ea) [0x7f3447dd811a] 2023-12-04T05:43:41.271 INFO:tasks.ceph.mds.b.smithi195.stderr: 15: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7f3447e6a991] 2023-12-04T05:43:41.271 INFO:tasks.ceph.mds.b.smithi195.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f3447578b43] 2023-12-04T05:43:41.273 INFO:tasks.ceph.mds.b.smithi195.stderr: 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f344760aa00] 2023-12-04T05:43:41.273 INFO:tasks.ceph.mds.b.smithi195.stderr: 2023-12-04T05:43:41.273 INFO:tasks.ceph.mds.b.smithi195.stderr: 0> 2023-12-04T05:43:41.212+0000 7f3441abe640 -1 *** Caught signal (Aborted) ** 2023-12-04T05:43:41.273 INFO:tasks.ceph.mds.b.smithi195.stderr: in thread 7f3441abe640 thread_name:ms_dispatch 2023-12-04T05:43:41.273 INFO:tasks.ceph.mds.b.smithi195.stderr: 2023-12-04T05:43:41.273 INFO:tasks.ceph.mds.b.smithi195.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:43:41.273 INFO:tasks.ceph.mds.b.smithi195.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f3447526520] 2023-12-04T05:43:41.274 INFO:tasks.ceph.mds.b.smithi195.stderr: 2: pthread_kill() 2023-12-04T05:43:41.274 INFO:tasks.ceph.mds.b.smithi195.stderr: 3: raise() 2023-12-04T05:43:41.274 INFO:tasks.ceph.mds.b.smithi195.stderr: 4: abort() 2023-12-04T05:43:41.274 INFO:tasks.ceph.mds.b.smithi195.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f3447bebef0] 2023-12-04T05:43:41.274 INFO:tasks.ceph.mds.b.smithi195.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f3447bec04a] 2023-12-04T05:43:41.274 INFO:tasks.ceph.mds.b.smithi195.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x560532b413e8] 2023-12-04T05:43:41.274 INFO:tasks.ceph.mds.b.smithi195.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x560532b36174] 2023-12-04T05:43:41.274 INFO:tasks.ceph.mds.b.smithi195.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x560532ae7033] 2023-12-04T05:43:41.274 INFO:tasks.ceph.mds.b.smithi195.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x560532a21dec] 2023-12-04T05:43:41.274 INFO:tasks.ceph.mds.b.smithi195.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x560532a28be8] 2023-12-04T05:43:41.274 INFO:tasks.ceph.mds.b.smithi195.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x560532a1ea91] 2023-12-04T05:43:41.274 INFO:tasks.ceph.mds.b.smithi195.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x560532a1f34c] 2023-12-04T05:43:41.274 INFO:tasks.ceph.mds.b.smithi195.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560532a0b837] 2023-12-04T05:43:41.274 INFO:tasks.ceph.mds.b.smithi195.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x56053298f5fb] 2023-12-04T05:43:41.275 INFO:tasks.ceph.mds.b.smithi195.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56053298facc] 2023-12-04T05:43:41.275 INFO:tasks.ceph.mds.b.smithi195.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x56053297a8e5] 2023-12-04T05:43:41.275 INFO:tasks.ceph.mds.b.smithi195.stderr: 18: (DispatchQueue::entry()+0x5ea) [0x7f3447dd811a] 2023-12-04T05:43:41.275 INFO:tasks.ceph.mds.b.smithi195.stderr: 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7f3447e6a991] 2023-12-04T05:43:41.275 INFO:tasks.ceph.mds.b.smithi195.stderr: 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f3447578b43] 2023-12-04T05:43:41.275 INFO:tasks.ceph.mds.b.smithi195.stderr: 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f344760aa00] 2023-12-04T05:43:41.275 INFO:tasks.ceph.mds.b.smithi195.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-04T05:43:41.275 INFO:tasks.ceph.mds.b.smithi195.stderr: 2023-12-04T05:43:41.547 INFO:tasks.ceph.mds.b.smithi195.stderr:daemon-helper: command crashed with signal 6 2023-12-04T05:43:47.711 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2023-12-04T05:43:47.711 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~41s 2023-12-04T05:43:54.424 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~7s 2023-12-04T05:43:54.425 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~48s 2023-12-04T05:44:00.401 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:44:00.404 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:44:01.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~13s 2023-12-04T05:44:01.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~54s 2023-12-04T05:44:07.851 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~20s 2023-12-04T05:44:07.852 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~61s 2023-12-04T05:44:14.565 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~27s 2023-12-04T05:44:14.565 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~68s 2023-12-04T05:44:21.281 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~34s 2023-12-04T05:44:21.281 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~74s 2023-12-04T05:44:27.995 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~40s 2023-12-04T05:44:27.996 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~81s 2023-12-04T05:44:30.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:44:30.533 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:44:34.710 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~47s 2023-12-04T05:44:34.710 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~88s 2023-12-04T05:44:41.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:44:41.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~54s 2023-12-04T05:44:41.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~95s 2023-12-04T05:44:41.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.463+0000 7feec041e640 1 Processor -- start 2023-12-04T05:44:41.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.463+0000 7feec041e640 1 -- start start 2023-12-04T05:44:41.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.463+0000 7feec041e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feeb80724f0 0x7feeb8158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:41.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.463+0000 7feec041e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feeb80c56f0 0x7feeb8140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:41.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.463+0000 7feec041e640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeb8158da0 con 0x7feeb8072110 2023-12-04T05:44:41.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.463+0000 7feec041e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feeb81583d0 con 0x7feeb80724f0 2023-12-04T05:44:41.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.463+0000 7feec041e640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeb80c6170 con 0x7feeb80c56f0 2023-12-04T05:44:41.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.463+0000 7feebd9b9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feeb80724f0 0x7feeb8158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:41.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.463+0000 7feebd9b9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feeb80724f0 0x7feeb8158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41150/0 (socket says 172.21.15.107:41150) 2023-12-04T05:44:41.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.463+0000 7feebd9b9640 1 -- 172.21.15.107:0/1366547172 learned_addr learned my addr 172.21.15.107:0/1366547172 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:44:41.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.463+0000 7feebe9bb640 1 --2- 172.21.15.107:0/1366547172 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feeb80c56f0 0x7feeb8140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:41.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.463+0000 7feebd9b9640 1 -- 172.21.15.107:0/1366547172 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feeb80c56f0 msgr2=0x7feeb8140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:41.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.463+0000 7feebd9b9640 1 --2- 172.21.15.107:0/1366547172 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feeb80c56f0 0x7feeb8140830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:41.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.463+0000 7feebd9b9640 1 -- 172.21.15.107:0/1366547172 >> v1:172.21.15.195:6789/0 conn(0x7feeb8072110 legacy=0x7feeb80c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:41.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.463+0000 7feebd9b9640 1 -- 172.21.15.107:0/1366547172 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb8157da0 con 0x7feeb80724f0 2023-12-04T05:44:41.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.463+0000 7feebd9b9640 1 --2- 172.21.15.107:0/1366547172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeb80724f0 0x7feeb8158860 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7feeb806b870 tx=0x7feea801c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deac9aedb13007b2 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:41.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feebd1b8640 1 -- 172.21.15.107:0/1366547172 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea80ab600 con 0x7feeb80724f0 2023-12-04T05:44:41.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feebd1b8640 1 -- 172.21.15.107:0/1366547172 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea80a7050 con 0x7feeb80724f0 2023-12-04T05:44:41.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feebd1b8640 1 -- 172.21.15.107:0/1366547172 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea80ab980 con 0x7feeb80724f0 2023-12-04T05:44:41.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feec041e640 1 -- 172.21.15.107:0/1366547172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeb80724f0 msgr2=0x7feeb8158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:41.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feec041e640 1 --2- 172.21.15.107:0/1366547172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeb80724f0 0x7feeb8158860 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7feeb806b870 tx=0x7feea801c4c0 comp rx=0 tx=0).stop 2023-12-04T05:44:41.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feec041e640 1 -- 172.21.15.107:0/1366547172 shutdown_connections 2023-12-04T05:44:41.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feec041e640 1 --2- 172.21.15.107:0/1366547172 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feeb80c56f0 0x7feeb8140830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:41.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feec041e640 1 --2- 172.21.15.107:0/1366547172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeb80724f0 0x7feeb8158860 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:41.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feec041e640 1 -- 172.21.15.107:0/1366547172 >> 172.21.15.107:0/1366547172 conn(0x7feeb80c7ff0 msgr2=0x7feeb80c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:44:41.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feec041e640 1 -- 172.21.15.107:0/1366547172 shutdown_connections 2023-12-04T05:44:41.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feec041e640 1 -- 172.21.15.107:0/1366547172 wait complete. 2023-12-04T05:44:41.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feec041e640 1 Processor -- start 2023-12-04T05:44:41.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feec041e640 1 -- start start 2023-12-04T05:44:41.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feec041e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feeb8072110 0x7feeb81f2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:41.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feec041e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeb80724f0 0x7feeb8158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:41.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feec041e640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feeb80c56f0 0x7feeb8156e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:41.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feec041e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb8152e80 con 0x7feeb80724f0 2023-12-04T05:44:41.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feec041e640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb8153160 con 0x7feeb80c56f0 2023-12-04T05:44:41.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feec041e640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeb8159580 con 0x7feeb8072110 2023-12-04T05:44:41.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feebd9b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeb80724f0 0x7feeb8158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:41.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feebd9b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeb80724f0 0x7feeb8158f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41158/0 (socket says 172.21.15.107:41158) 2023-12-04T05:44:41.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feebd9b9640 1 -- 172.21.15.107:0/1321340214 learned_addr learned my addr 172.21.15.107:0/1321340214 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:44:41.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feebe1ba640 1 --2- 172.21.15.107:0/1321340214 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feeb8072110 0x7feeb81f2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:41.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feebd9b9640 1 -- 172.21.15.107:0/1321340214 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feeb8072110 msgr2=0x7feeb81f2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:41.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feebd9b9640 1 --2- 172.21.15.107:0/1321340214 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feeb8072110 0x7feeb81f2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:41.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feebd9b9640 1 -- 172.21.15.107:0/1321340214 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feeb80c56f0 msgr2=0x7feeb8156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:41.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feebd9b9640 1 --2- 172.21.15.107:0/1321340214 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feeb80c56f0 0x7feeb8156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:41.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feebd9b9640 1 -- 172.21.15.107:0/1321340214 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea80094d0 con 0x7feeb80724f0 2023-12-04T05:44:41.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feebd9b9640 1 --2- 172.21.15.107:0/1321340214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeb80724f0 0x7feeb8158f60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7feea80ae040 tx=0x7feea8019260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:41.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feeaeffd640 1 -- 172.21.15.107:0/1321340214 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea80b4640 con 0x7feeb80724f0 2023-12-04T05:44:41.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feeaeffd640 1 -- 172.21.15.107:0/1321340214 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea80b47c0 con 0x7feeb80724f0 2023-12-04T05:44:41.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feec041e640 1 -- 172.21.15.107:0/1321340214 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeb8159860 con 0x7feeb80724f0 2023-12-04T05:44:41.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feeaeffd640 1 -- 172.21.15.107:0/1321340214 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea80b4b70 con 0x7feeb80724f0 2023-12-04T05:44:41.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.467+0000 7feec041e640 1 -- 172.21.15.107:0/1321340214 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeb8159e20 con 0x7feeb80724f0 2023-12-04T05:44:41.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.471+0000 7feec041e640 1 -- 172.21.15.107:0/1321340214 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeb8067870 con 0x7feeb80724f0 2023-12-04T05:44:41.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.471+0000 7feeaeffd640 1 -- 172.21.15.107:0/1321340214 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7feea80b2070 con 0x7feeb80724f0 2023-12-04T05:44:41.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.471+0000 7feeaeffd640 1 --2- 172.21.15.107:0/1321340214 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fee9c0668d0 0x7fee9c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:41.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.471+0000 7feeaeffd640 1 -- 172.21.15.107:0/1321340214 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5997+0+0 (secure 0 0 0) 0x7feea811e490 con 0x7feeb80724f0 2023-12-04T05:44:41.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.471+0000 7feebe1ba640 1 --2- 172.21.15.107:0/1321340214 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fee9c0668d0 0x7fee9c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:41.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.471+0000 7feebe1ba640 1 --2- 172.21.15.107:0/1321340214 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fee9c0668d0 0x7fee9c068d90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7feeb4003d60 tx=0x7feeb4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:41.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.475+0000 7feeaeffd640 1 -- 172.21.15.107:0/1321340214 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7feea80eb5b0 con 0x7feeb80724f0 2023-12-04T05:44:41.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.663+0000 7feec041e640 1 -- 172.21.15.107:0/1321340214 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feeb8152350 con 0x7feeb80724f0 2023-12-04T05:44:41.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.663+0000 7feeaeffd640 1 -- 172.21.15.107:0/1321340214 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5825 (secure 0 0 0) 0x7feea80eebd0 con 0x7feeb80724f0 2023-12-04T05:44:41.666 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:44:41.666 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:43:59.753818+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4747,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:active","state_seq":45,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4747":{"gid":4747,"name":"b","rank":1,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.195:6836/2222928136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2222928136},{"type":"v1","addr":"172.21.15.195:6836","nonce":2222928136}]},"join_fscid":-1,"laggy_since":"2023-12-04T05:43:59.753818+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:44:41.667 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 30 2023-12-04T05:44:41.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.667+0000 7feec041e640 1 -- 172.21.15.107:0/1321340214 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fee9c0668d0 msgr2=0x7fee9c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:41.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.667+0000 7feec041e640 1 --2- 172.21.15.107:0/1321340214 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fee9c0668d0 0x7fee9c068d90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7feeb4003d60 tx=0x7feeb4009040 comp rx=0 tx=0).stop 2023-12-04T05:44:41.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.667+0000 7feec041e640 1 -- 172.21.15.107:0/1321340214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeb80724f0 msgr2=0x7feeb8158f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:41.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.667+0000 7feec041e640 1 --2- 172.21.15.107:0/1321340214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeb80724f0 0x7feeb8158f60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7feea80ae040 tx=0x7feea8019260 comp rx=0 tx=0).stop 2023-12-04T05:44:41.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.667+0000 7feec041e640 1 -- 172.21.15.107:0/1321340214 shutdown_connections 2023-12-04T05:44:41.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.667+0000 7feec041e640 1 --2- 172.21.15.107:0/1321340214 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fee9c0668d0 0x7fee9c068d90 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:41.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.667+0000 7feec041e640 1 --2- 172.21.15.107:0/1321340214 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7feeb80c56f0 0x7feeb8156e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:41.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.667+0000 7feec041e640 1 --2- 172.21.15.107:0/1321340214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feeb80724f0 0x7feeb8158f60 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:41.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.667+0000 7feec041e640 1 --2- 172.21.15.107:0/1321340214 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7feeb8072110 0x7feeb81f2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:41.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.667+0000 7feec041e640 1 -- 172.21.15.107:0/1321340214 >> 172.21.15.107:0/1321340214 conn(0x7feeb80c7ff0 msgr2=0x7feeb8162f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:44:41.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.667+0000 7feec041e640 1 -- 172.21.15.107:0/1321340214 shutdown_connections 2023-12-04T05:44:41.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.667+0000 7feec041e640 1 -- 172.21.15.107:0/1321340214 wait complete. 2023-12-04T05:44:41.685 INFO:tasks.mds_thrash.fs.[cephfs]:thrashing max_mds: 5 -> 2 2023-12-04T05:44:41.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2023-12-04T05:44:41.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.803+0000 7f8124d7d640 1 Processor -- start 2023-12-04T05:44:41.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.803+0000 7f8124d7d640 1 -- start start 2023-12-04T05:44:41.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f8124d7d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8120072110 0x7f8120158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:41.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f8124d7d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81200c56f0 0x7f8120140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:41.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f8124d7d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8120158da0 con 0x7f81200724f0 2023-12-04T05:44:41.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f8124d7d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81201583d0 con 0x7f8120072110 2023-12-04T05:44:41.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f8124d7d640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81200c6170 con 0x7f81200c56f0 2023-12-04T05:44:41.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f811e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8120072110 0x7f8120158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:41.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f811e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8120072110 0x7f8120158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41172/0 (socket says 172.21.15.107:41172) 2023-12-04T05:44:41.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f811e59c640 1 -- 172.21.15.107:0/1158237351 learned_addr learned my addr 172.21.15.107:0/1158237351 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:44:41.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f811ed9d640 1 --2- 172.21.15.107:0/1158237351 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81200c56f0 0x7f8120140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:41.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f811e59c640 1 -- 172.21.15.107:0/1158237351 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81200c56f0 msgr2=0x7f8120140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:41.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f811e59c640 1 --2- 172.21.15.107:0/1158237351 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81200c56f0 0x7f8120140830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:41.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f811e59c640 1 -- 172.21.15.107:0/1158237351 >> v1:172.21.15.195:6789/0 conn(0x7f81200724f0 legacy=0x7f81200c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:41.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f811e59c640 1 -- 172.21.15.107:0/1158237351 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8120157da0 con 0x7f8120072110 2023-12-04T05:44:41.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f811e59c640 1 --2- 172.21.15.107:0/1158237351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8120072110 0x7f8120158860 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f812006b8c0 tx=0x7f811001c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35dff240abb4045a server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:41.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f811dd9b640 1 -- 172.21.15.107:0/1158237351 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81100ad640 con 0x7f8120072110 2023-12-04T05:44:41.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f811dd9b640 1 -- 172.21.15.107:0/1158237351 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81100ad7a0 con 0x7f8120072110 2023-12-04T05:44:41.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f811dd9b640 1 -- 172.21.15.107:0/1158237351 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81100adb30 con 0x7f8120072110 2023-12-04T05:44:41.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f8124d7d640 1 -- 172.21.15.107:0/1158237351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8120072110 msgr2=0x7f8120158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:41.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f8124d7d640 1 --2- 172.21.15.107:0/1158237351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8120072110 0x7f8120158860 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f812006b8c0 tx=0x7f811001c4e0 comp rx=0 tx=0).stop 2023-12-04T05:44:41.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f8124d7d640 1 -- 172.21.15.107:0/1158237351 shutdown_connections 2023-12-04T05:44:41.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f8124d7d640 1 --2- 172.21.15.107:0/1158237351 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81200c56f0 0x7f8120140830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:41.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f8124d7d640 1 --2- 172.21.15.107:0/1158237351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8120072110 0x7f8120158860 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:41.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.807+0000 7f8124d7d640 1 -- 172.21.15.107:0/1158237351 >> 172.21.15.107:0/1158237351 conn(0x7f81200c7ff0 msgr2=0x7f81200c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:44:41.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f8124d7d640 1 -- 172.21.15.107:0/1158237351 shutdown_connections 2023-12-04T05:44:41.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f8124d7d640 1 -- 172.21.15.107:0/1158237351 wait complete. 2023-12-04T05:44:41.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f8124d7d640 1 Processor -- start 2023-12-04T05:44:41.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f8124d7d640 1 -- start start 2023-12-04T05:44:41.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f8124d7d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8120072110 0x7f81201f4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:41.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f8124d7d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81200724f0 0x7f81201f0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:41.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f8124d7d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81200c56f0 0x7f812013cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:41.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f8124d7d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812015a870 con 0x7f81200c56f0 2023-12-04T05:44:41.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f8124d7d640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81201f0d70 con 0x7f8120072110 2023-12-04T05:44:41.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f8124d7d640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f812015c190 con 0x7f81200724f0 2023-12-04T05:44:41.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f811ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81200c56f0 0x7f812013cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:41.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f811ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81200c56f0 0x7f812013cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41182/0 (socket says 172.21.15.107:41182) 2023-12-04T05:44:41.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f811ed9d640 1 -- 172.21.15.107:0/2629859094 learned_addr learned my addr 172.21.15.107:0/2629859094 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:44:41.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f8115d9b640 1 --2- 172.21.15.107:0/2629859094 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81200724f0 0x7f81201f0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:41.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f811e59c640 1 --2- 172.21.15.107:0/2629859094 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8120072110 0x7f81201f4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:41.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f811ed9d640 1 -- 172.21.15.107:0/2629859094 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81200724f0 msgr2=0x7f81201f0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:41.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f811ed9d640 1 --2- 172.21.15.107:0/2629859094 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81200724f0 0x7f81201f0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:41.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f811ed9d640 1 -- 172.21.15.107:0/2629859094 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8120072110 msgr2=0x7f81201f4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:41.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f811ed9d640 1 --2- 172.21.15.107:0/2629859094 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8120072110 0x7f81201f4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:41.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f811ed9d640 1 -- 172.21.15.107:0/2629859094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81100094d0 con 0x7f81200c56f0 2023-12-04T05:44:41.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f811ed9d640 1 --2- 172.21.15.107:0/2629859094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81200c56f0 0x7f812013cb40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f811800cc30 tx=0x7f811800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:41.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f81177fe640 1 -- 172.21.15.107:0/2629859094 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811800ce40 con 0x7f81200c56f0 2023-12-04T05:44:41.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f81177fe640 1 -- 172.21.15.107:0/2629859094 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81180039a0 con 0x7f81200c56f0 2023-12-04T05:44:41.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f81177fe640 1 -- 172.21.15.107:0/2629859094 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8118003d30 con 0x7f81200c56f0 2023-12-04T05:44:41.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f8124d7d640 1 -- 172.21.15.107:0/2629859094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f812015bbd0 con 0x7f81200c56f0 2023-12-04T05:44:41.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.811+0000 7f8124d7d640 1 -- 172.21.15.107:0/2629859094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81201f3e40 con 0x7f81200c56f0 2023-12-04T05:44:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.815+0000 7f8124d7d640 1 -- 172.21.15.107:0/2629859094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81200c6690 con 0x7f81200c56f0 2023-12-04T05:44:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.815+0000 7f81177fe640 1 -- 172.21.15.107:0/2629859094 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f8118003e90 con 0x7f81200c56f0 2023-12-04T05:44:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.815+0000 7f81177fe640 1 --2- 172.21.15.107:0/2629859094 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f81000668d0 0x7f8100068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.815+0000 7f81177fe640 1 -- 172.21.15.107:0/2629859094 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f8118084930 con 0x7f81200c56f0 2023-12-04T05:44:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.815+0000 7f811e59c640 1 --2- 172.21.15.107:0/2629859094 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f81000668d0 0x7f8100068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.815+0000 7f811e59c640 1 --2- 172.21.15.107:0/2629859094 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f81000668d0 0x7f8100068d90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f81100af000 tx=0x7f8110008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:41.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:41.819+0000 7f81177fe640 1 -- 172.21.15.107:0/2629859094 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8118051a50 con 0x7f81200c56f0 2023-12-04T05:44:42.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:42.007+0000 7f8124d7d640 1 -- 172.21.15.107:0/2629859094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f8120067870 con 0x7f81200c56f0 2023-12-04T05:44:42.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:42.891+0000 7f81177fe640 1 -- 172.21.15.107:0/2629859094 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v31) v1 ==== 105+0+0 (secure 0 0 0) 0x7f811801d070 con 0x7f81200c56f0 2023-12-04T05:44:42.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:42.895+0000 7f8124d7d640 1 -- 172.21.15.107:0/2629859094 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f81000668d0 msgr2=0x7f8100068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:42.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:42.895+0000 7f8124d7d640 1 --2- 172.21.15.107:0/2629859094 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f81000668d0 0x7f8100068d90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f81100af000 tx=0x7f8110008040 comp rx=0 tx=0).stop 2023-12-04T05:44:42.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:42.895+0000 7f8124d7d640 1 -- 172.21.15.107:0/2629859094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81200c56f0 msgr2=0x7f812013cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:42.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:42.895+0000 7f8124d7d640 1 --2- 172.21.15.107:0/2629859094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81200c56f0 0x7f812013cb40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f811800cc30 tx=0x7f811800a5c0 comp rx=0 tx=0).stop 2023-12-04T05:44:42.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:42.895+0000 7f8124d7d640 1 -- 172.21.15.107:0/2629859094 shutdown_connections 2023-12-04T05:44:42.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:42.895+0000 7f8124d7d640 1 --2- 172.21.15.107:0/2629859094 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f81000668d0 0x7f8100068d90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:42.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:42.895+0000 7f8124d7d640 1 --2- 172.21.15.107:0/2629859094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81200c56f0 0x7f812013cb40 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:42.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:42.895+0000 7f8124d7d640 1 --2- 172.21.15.107:0/2629859094 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81200724f0 0x7f81201f0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:42.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:42.895+0000 7f8124d7d640 1 --2- 172.21.15.107:0/2629859094 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8120072110 0x7f81201f4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:42.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:42.895+0000 7f8124d7d640 1 -- 172.21.15.107:0/2629859094 >> 172.21.15.107:0/2629859094 conn(0x7f81200c7ff0 msgr2=0x7f8120157420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:44:42.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:42.895+0000 7f8124d7d640 1 -- 172.21.15.107:0/2629859094 shutdown_connections 2023-12-04T05:44:42.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:42.895+0000 7f8124d7d640 1 -- 172.21.15.107:0/2629859094 wait complete. 2023-12-04T05:44:42.920 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-12-04T05:44:42.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:44:43.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.043+0000 7f417d115640 1 Processor -- start 2023-12-04T05:44:43.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.043+0000 7f417d115640 1 -- start start 2023-12-04T05:44:43.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.047+0000 7f417d115640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f41780c4a80 0x7f41780c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:43.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.047+0000 7f417d115640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f417806e660 con 0x7f4178157180 2023-12-04T05:44:43.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.047+0000 7f417d115640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4178156cf0 con 0x7f41780c46a0 2023-12-04T05:44:43.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.047+0000 7f417d115640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4178156490 con 0x7f41780c4a80 2023-12-04T05:44:43.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.047+0000 7f416e59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4178157180 0x7f41780c50e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57988/0 (socket says 172.21.15.107:57988) 2023-12-04T05:44:43.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.047+0000 7f417759e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f41780c4a80 0x7f41780c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:43.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.047+0000 7f416e59c640 1 -- 172.21.15.107:0/460252374 learned_addr learned my addr 172.21.15.107:0/460252374 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:44:43.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.047+0000 7f417759e640 1 -- 172.21.15.107:0/460252374 >> v1:172.21.15.195:6789/0 conn(0x7f41780c46a0 legacy=0x7f41780c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:43.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.047+0000 7f417659c640 1 -- 172.21.15.107:0/460252374 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755137011 0 0) 0x7f4160001120 con 0x7f4178157180 2023-12-04T05:44:43.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.047+0000 7f417759e640 1 -- 172.21.15.107:0/460252374 >> v1:172.21.15.107:6789/0 conn(0x7f4178157180 legacy=0x7f41780c50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:43.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.047+0000 7f417759e640 1 -- 172.21.15.107:0/460252374 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417806e0a0 con 0x7f41780c4a80 2023-12-04T05:44:43.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.047+0000 7f417759e640 1 --2- 172.21.15.107:0/460252374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41780c4a80 0x7f41780c8b10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f417000aea0 tx=0x7f417001da10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=875c842b786350d server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:43.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f417659c640 1 -- 172.21.15.107:0/460252374 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417000a2d0 con 0x7f41780c4a80 2023-12-04T05:44:43.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f417659c640 1 -- 172.21.15.107:0/460252374 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41700a66f0 con 0x7f41780c4a80 2023-12-04T05:44:43.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f417659c640 1 -- 172.21.15.107:0/460252374 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41700a6a80 con 0x7f41780c4a80 2023-12-04T05:44:43.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f417d115640 1 -- 172.21.15.107:0/460252374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41780c4a80 msgr2=0x7f41780c8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:43.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f417d115640 1 --2- 172.21.15.107:0/460252374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41780c4a80 0x7f41780c8b10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f417000aea0 tx=0x7f417001da10 comp rx=0 tx=0).stop 2023-12-04T05:44:43.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f417d115640 1 -- 172.21.15.107:0/460252374 shutdown_connections 2023-12-04T05:44:43.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f417d115640 1 --2- 172.21.15.107:0/460252374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41780c4a80 0x7f41780c8b10 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:43.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f417d115640 1 -- 172.21.15.107:0/460252374 >> 172.21.15.107:0/460252374 conn(0x7f41780c7ff0 msgr2=0x7f41780c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:44:43.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f417d115640 1 -- 172.21.15.107:0/460252374 shutdown_connections 2023-12-04T05:44:43.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f417d115640 1 -- 172.21.15.107:0/460252374 wait complete. 2023-12-04T05:44:43.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f417d115640 1 Processor -- start 2023-12-04T05:44:43.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f417d115640 1 -- start start 2023-12-04T05:44:43.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f417d115640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41780c46a0 0x7f4178071660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:43.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f417d115640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41780c4a80 0x7f41780c33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:43.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f417d115640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4178157180 0x7f4178146750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:43.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f417d115640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41781462c0 con 0x7f41780c4a80 2023-12-04T05:44:43.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f417d115640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417814aef0 con 0x7f41780c46a0 2023-12-04T05:44:43.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f417d115640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f417814b1b0 con 0x7f4178157180 2023-12-04T05:44:43.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f416e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41780c4a80 0x7f41780c33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:43.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f416e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41780c4a80 0x7f41780c33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41198/0 (socket says 172.21.15.107:41198) 2023-12-04T05:44:43.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f416e59c640 1 -- 172.21.15.107:0/1174353654 learned_addr learned my addr 172.21.15.107:0/1174353654 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:44:43.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f417759e640 1 --2- 172.21.15.107:0/1174353654 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4178157180 0x7f4178146750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:43.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f4176d9d640 1 --2- 172.21.15.107:0/1174353654 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41780c46a0 0x7f4178071660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:43.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f416e59c640 1 -- 172.21.15.107:0/1174353654 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4178157180 msgr2=0x7f4178146750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:43.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f416e59c640 1 --2- 172.21.15.107:0/1174353654 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4178157180 0x7f4178146750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:43.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f416e59c640 1 -- 172.21.15.107:0/1174353654 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41780c46a0 msgr2=0x7f4178071660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:43.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f416e59c640 1 --2- 172.21.15.107:0/1174353654 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41780c46a0 0x7f4178071660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:43.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f416e59c640 1 -- 172.21.15.107:0/1174353654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417000ab80 con 0x7f41780c4a80 2023-12-04T05:44:43.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f417759e640 1 --2- 172.21.15.107:0/1174353654 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4178157180 0x7f4178146750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:44:43.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f4176d9d640 1 --2- 172.21.15.107:0/1174353654 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41780c46a0 0x7f4178071660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:44:43.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f416e59c640 1 --2- 172.21.15.107:0/1174353654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41780c4a80 0x7f41780c33f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f416000c480 tx=0x7f416000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:43.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f416ffff640 1 -- 172.21.15.107:0/1174353654 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41600074e0 con 0x7f41780c4a80 2023-12-04T05:44:43.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f416ffff640 1 -- 172.21.15.107:0/1174353654 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4160007660 con 0x7f41780c4a80 2023-12-04T05:44:43.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f417d115640 1 -- 172.21.15.107:0/1174353654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4178145c70 con 0x7f41780c4a80 2023-12-04T05:44:43.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f416ffff640 1 -- 172.21.15.107:0/1174353654 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4160004a30 con 0x7f41780c4a80 2023-12-04T05:44:43.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.051+0000 7f417d115640 1 -- 172.21.15.107:0/1174353654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4178145970 con 0x7f41780c4a80 2023-12-04T05:44:43.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.055+0000 7f417d115640 1 -- 172.21.15.107:0/1174353654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4178067870 con 0x7f41780c4a80 2023-12-04T05:44:43.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.055+0000 7f416ffff640 1 -- 172.21.15.107:0/1174353654 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4160004b90 con 0x7f41780c4a80 2023-12-04T05:44:43.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.055+0000 7f416ffff640 1 --2- 172.21.15.107:0/1174353654 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f41580668d0 0x7f4158068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:43.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.055+0000 7f416ffff640 1 -- 172.21.15.107:0/1174353654 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f41600838f0 con 0x7f41780c4a80 2023-12-04T05:44:43.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.055+0000 7f4176d9d640 1 --2- 172.21.15.107:0/1174353654 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f41580668d0 0x7f4158068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:43.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.055+0000 7f4176d9d640 1 --2- 172.21.15.107:0/1174353654 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f41580668d0 0x7f4158068d90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4168003e40 tx=0x7f4168009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:43.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.059+0000 7f416ffff640 1 -- 172.21.15.107:0/1174353654 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4160054490 con 0x7f41780c4a80 2023-12-04T05:44:43.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.247+0000 7f417d115640 1 -- 172.21.15.107:0/1174353654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f417806e660 con 0x7f41780c4a80 2023-12-04T05:44:43.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.247+0000 7f416ffff640 1 -- 172.21.15.107:0/1174353654 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5827 (secure 0 0 0) 0x7f417806e660 con 0x7f41780c4a80 2023-12-04T05:44:43.250 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:44:43.250 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:44:42.900817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4747,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:stopping","state_seq":45,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4747":{"gid":4747,"name":"b","rank":1,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.195:6836/2222928136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2222928136},{"type":"v1","addr":"172.21.15.195:6836","nonce":2222928136}]},"join_fscid":-1,"laggy_since":"2023-12-04T05:43:59.753818+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:44:43.251 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 32 2023-12-04T05:44:43.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.251+0000 7f417d115640 1 -- 172.21.15.107:0/1174353654 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f41580668d0 msgr2=0x7f4158068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:43.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.251+0000 7f417d115640 1 --2- 172.21.15.107:0/1174353654 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f41580668d0 0x7f4158068d90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4168003e40 tx=0x7f4168009040 comp rx=0 tx=0).stop 2023-12-04T05:44:43.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.251+0000 7f417d115640 1 -- 172.21.15.107:0/1174353654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41780c4a80 msgr2=0x7f41780c33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:43.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.251+0000 7f417d115640 1 --2- 172.21.15.107:0/1174353654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41780c4a80 0x7f41780c33f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f416000c480 tx=0x7f416000c940 comp rx=0 tx=0).stop 2023-12-04T05:44:43.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.251+0000 7f417d115640 1 -- 172.21.15.107:0/1174353654 shutdown_connections 2023-12-04T05:44:43.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.251+0000 7f417d115640 1 --2- 172.21.15.107:0/1174353654 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f41580668d0 0x7f4158068d90 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:43.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.251+0000 7f417d115640 1 --2- 172.21.15.107:0/1174353654 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4178157180 0x7f4178146750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:43.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.251+0000 7f417d115640 1 --2- 172.21.15.107:0/1174353654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41780c4a80 0x7f41780c33f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:43.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.251+0000 7f417d115640 1 --2- 172.21.15.107:0/1174353654 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41780c46a0 0x7f4178071660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:43.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.251+0000 7f417d115640 1 -- 172.21.15.107:0/1174353654 >> 172.21.15.107:0/1174353654 conn(0x7f41780c7ff0 msgr2=0x7f4178153e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:44:43.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.251+0000 7f417d115640 1 -- 172.21.15.107:0/1174353654 shutdown_connections 2023-12-04T05:44:43.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:43.251+0000 7f417d115640 1 -- 172.21.15.107:0/1174353654 wait complete. 2023-12-04T05:44:43.268 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-12-04T05:44:43.269 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 32, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-12-04T05:39:59.032019+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 32, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4247": { "addr": "172.21.15.107:6839/993693883", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6838", "nonce": 993693883, "type": "v2" }, { "addr": "172.21.15.107:6839", "nonce": 993693883, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 18, "join_fscid": -1, "name": "a", "rank": 4, "state": "up:stopping", "state_seq": 45 }, "gid_4260": { "addr": "172.21.15.107:6836/3444851515", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 3444851515, "type": "v2" }, { "addr": "172.21.15.107:6836", "nonce": 3444851515, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4260, "incarnation": 14, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 9 }, "gid_4263": { "addr": "172.21.15.107:6837/2210527429", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6835", "nonce": 2210527429, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 2210527429, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4263, "incarnation": 12, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 7 }, "gid_4288": { "addr": "172.21.15.195:6837/3412865477", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6835", "nonce": 3412865477, "type": "v2" }, { "addr": "172.21.15.195:6837", "nonce": 3412865477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4747": { "addr": "172.21.15.195:6836/2222928136", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2222928136, "type": "v2" }, { "addr": "172.21.15.195:6836", "nonce": 2222928136, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4747, "incarnation": 25, "join_fscid": -1, "laggy_since": "2023-12-04T05:43:59.753818+0000", "name": "b", "rank": 1, "state": "up:active", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 25, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-12-04T05:44:42.900817+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 4747, "mds_2": 4263, "mds_3": 4260, "mds_4": 4247 } } } ], "standbys": [] } 2023-12-04T05:44:45.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:44:45.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.395+0000 7ff14fd94640 1 Processor -- start 2023-12-04T05:44:45.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14fd94640 1 -- start start 2023-12-04T05:44:45.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14fd94640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff1480cd8c0 0x7ff1480c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:45.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14fd94640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff148072e50 con 0x7ff1480c8990 2023-12-04T05:44:45.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14fd94640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14815c100 con 0x7ff1480cdca0 2023-12-04T05:44:45.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14fd94640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff148072960 con 0x7ff1480cd8c0 2023-12-04T05:44:45.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14e331640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff1480c8990 0x7ff148151970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57998/0 (socket says 172.21.15.107:57998) 2023-12-04T05:44:45.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14e331640 1 -- 172.21.15.107:0/2982807242 learned_addr learned my addr 172.21.15.107:0/2982807242 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:44:45.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14db30640 1 --2- 172.21.15.107:0/2982807242 >> v2:172.21.15.195:3300/0 conn(0x7ff1480cd8c0 0x7ff1480c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:45.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14cb2e640 1 -- 172.21.15.107:0/2982807242 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310364065 0 0) 0x7ff148072e50 con 0x7ff1480c8990 2023-12-04T05:44:45.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14cb2e640 1 -- 172.21.15.107:0/2982807242 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff134003530 con 0x7ff1480c8990 2023-12-04T05:44:45.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14cb2e640 1 -- 172.21.15.107:0/2982807242 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2753466540 0 0) 0x7ff134003530 con 0x7ff1480c8990 2023-12-04T05:44:45.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14cb2e640 1 -- 172.21.15.107:0/2982807242 >> v2:172.21.15.195:3300/0 conn(0x7ff1480cd8c0 msgr2=0x7ff1480c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:45.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14cb2e640 1 --2- 172.21.15.107:0/2982807242 >> v2:172.21.15.195:3300/0 conn(0x7ff1480cd8c0 0x7ff1480c80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:45.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14cb2e640 1 -- 172.21.15.107:0/2982807242 >> v1:172.21.15.195:6789/0 conn(0x7ff1480cdca0 legacy=0x7ff148072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:45.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14cb2e640 1 -- 172.21.15.107:0/2982807242 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1480c8630 con 0x7ff1480c8990 2023-12-04T05:44:45.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14cb2e640 1 -- 172.21.15.107:0/2982807242 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7ff144001680 con 0x7ff1480c8990 2023-12-04T05:44:45.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14db30640 1 --2- 172.21.15.107:0/2982807242 >> v2:172.21.15.195:3300/0 conn(0x7ff1480cd8c0 0x7ff1480c80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:44:45.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14cb2e640 1 -- 172.21.15.107:0/2982807242 >> v1:172.21.15.107:6789/0 conn(0x7ff1480c8990 legacy=0x7ff148151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:45.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14cb2e640 1 --2- 172.21.15.107:0/2982807242 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff134003980 0x7ff134003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:45.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14cb2e640 1 --2- 172.21.15.107:0/2982807242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1340047d0 0x7ff134004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:45.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14cb2e640 1 --2- 172.21.15.107:0/2982807242 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff134005190 0x7ff13400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:45.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14cb2e640 1 -- 172.21.15.107:0/2982807242 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff148072960 con 0x7ff1340047d0 2023-12-04T05:44:45.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14cb2e640 1 -- 172.21.15.107:0/2982807242 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13400c0c0 con 0x7ff134003980 2023-12-04T05:44:45.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14cb2e640 1 -- 172.21.15.107:0/2982807242 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff13400c280 con 0x7ff134005190 2023-12-04T05:44:45.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14d32f640 1 --2- 172.21.15.107:0/2982807242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1340047d0 0x7ff134004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:45.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14db30640 1 --2- 172.21.15.107:0/2982807242 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff134003980 0x7ff134003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:45.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14e331640 1 --2- 172.21.15.107:0/2982807242 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff134005190 0x7ff13400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:45.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14d32f640 1 -- 172.21.15.107:0/2982807242 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff134005190 msgr2=0x7ff13400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:45.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14d32f640 1 --2- 172.21.15.107:0/2982807242 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff134005190 0x7ff13400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:45.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14d32f640 1 -- 172.21.15.107:0/2982807242 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff134003980 msgr2=0x7ff134003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:45.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14d32f640 1 --2- 172.21.15.107:0/2982807242 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff134003980 0x7ff134003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:45.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14d32f640 1 -- 172.21.15.107:0/2982807242 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff134003530 con 0x7ff1340047d0 2023-12-04T05:44:45.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14e331640 1 --2- 172.21.15.107:0/2982807242 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff134005190 0x7ff13400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:44:45.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.399+0000 7ff14e331640 1 -- 172.21.15.107:0/2982807242 reap_dead start 2023-12-04T05:44:45.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14d32f640 1 --2- 172.21.15.107:0/2982807242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1340047d0 0x7ff134004c50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff138001780 tx=0x7ff13801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a853d8410fe87fa4 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:45.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14cb2e640 1 -- 172.21.15.107:0/2982807242 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1380052d0 con 0x7ff1340047d0 2023-12-04T05:44:45.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14cb2e640 1 -- 172.21.15.107:0/2982807242 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1380a7050 con 0x7ff1340047d0 2023-12-04T05:44:45.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14cb2e640 1 -- 172.21.15.107:0/2982807242 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff138003600 con 0x7ff1340047d0 2023-12-04T05:44:45.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14fd94640 1 -- 172.21.15.107:0/2982807242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1340047d0 msgr2=0x7ff134004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:45.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14fd94640 1 --2- 172.21.15.107:0/2982807242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1340047d0 0x7ff134004c50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff138001780 tx=0x7ff13801c3e0 comp rx=0 tx=0).stop 2023-12-04T05:44:45.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14fd94640 1 -- 172.21.15.107:0/2982807242 shutdown_connections 2023-12-04T05:44:45.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14fd94640 1 --2- 172.21.15.107:0/2982807242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1340047d0 0x7ff134004c50 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:45.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14fd94640 1 -- 172.21.15.107:0/2982807242 >> 172.21.15.107:0/2982807242 conn(0x7ff14806e280 msgr2=0x7ff1480663a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:44:45.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14fd94640 1 -- 172.21.15.107:0/2982807242 shutdown_connections 2023-12-04T05:44:45.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14fd94640 1 -- 172.21.15.107:0/2982807242 wait complete. 2023-12-04T05:44:45.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14fd94640 1 Processor -- start 2023-12-04T05:44:45.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14fd94640 1 -- start start 2023-12-04T05:44:45.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14fd94640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1340047d0 0x7ff14815a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:45.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14fd94640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff148151190 0x7ff14806ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:45.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14fd94640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff14806c080 0x7ff148140ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:45.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14fd94640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1481f2ef0 con 0x7ff1340047d0 2023-12-04T05:44:45.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14fd94640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1480c9ac0 con 0x7ff14806c080 2023-12-04T05:44:45.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14fd94640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14814cfc0 con 0x7ff148151190 2023-12-04T05:44:45.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14db30640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1340047d0 0x7ff14815a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:45.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14db30640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1340047d0 0x7ff14815a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41224/0 (socket says 172.21.15.107:41224) 2023-12-04T05:44:45.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14db30640 1 -- 172.21.15.107:0/3871931521 learned_addr learned my addr 172.21.15.107:0/3871931521 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:44:45.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14e331640 1 --2- 172.21.15.107:0/3871931521 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff14806c080 0x7ff148140ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:45.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.403+0000 7ff14d32f640 1 --2- 172.21.15.107:0/3871931521 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff148151190 0x7ff14806ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:45.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.407+0000 7ff14db30640 1 -- 172.21.15.107:0/3871931521 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff148151190 msgr2=0x7ff14806ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:45.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.407+0000 7ff14db30640 1 --2- 172.21.15.107:0/3871931521 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff148151190 0x7ff14806ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:45.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.407+0000 7ff14db30640 1 -- 172.21.15.107:0/3871931521 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff14806c080 msgr2=0x7ff148140ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:45.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.407+0000 7ff14db30640 1 --2- 172.21.15.107:0/3871931521 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff14806c080 0x7ff148140ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:45.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.407+0000 7ff14db30640 1 -- 172.21.15.107:0/3871931521 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14814d2a0 con 0x7ff1340047d0 2023-12-04T05:44:45.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.407+0000 7ff14d32f640 1 --2- 172.21.15.107:0/3871931521 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff148151190 0x7ff14806ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:44:45.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.407+0000 7ff14db30640 1 --2- 172.21.15.107:0/3871931521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1340047d0 0x7ff14815a3f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff140004700 tx=0x7ff140007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:45.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.407+0000 7ff13e7fc640 1 -- 172.21.15.107:0/3871931521 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff140014610 con 0x7ff1340047d0 2023-12-04T05:44:45.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.407+0000 7ff13e7fc640 1 -- 172.21.15.107:0/3871931521 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14000f050 con 0x7ff1340047d0 2023-12-04T05:44:45.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.407+0000 7ff14fd94640 1 -- 172.21.15.107:0/3871931521 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff14814d580 con 0x7ff1340047d0 2023-12-04T05:44:45.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.407+0000 7ff13e7fc640 1 -- 172.21.15.107:0/3871931521 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1400149b0 con 0x7ff1340047d0 2023-12-04T05:44:45.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.407+0000 7ff14fd94640 1 -- 172.21.15.107:0/3871931521 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14806c950 con 0x7ff1340047d0 2023-12-04T05:44:45.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.407+0000 7ff14fd94640 1 -- 172.21.15.107:0/3871931521 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff148067ec0 con 0x7ff1340047d0 2023-12-04T05:44:45.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.407+0000 7ff13e7fc640 1 -- 172.21.15.107:0/3871931521 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff140014d20 con 0x7ff1340047d0 2023-12-04T05:44:45.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.407+0000 7ff13e7fc640 1 --2- 172.21.15.107:0/3871931521 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff12c0668d0 0x7ff12c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:45.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.407+0000 7ff13e7fc640 1 -- 172.21.15.107:0/3871931521 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5997+0+0 (secure 0 0 0) 0x7ff140085230 con 0x7ff1340047d0 2023-12-04T05:44:45.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.407+0000 7ff14d32f640 1 --2- 172.21.15.107:0/3871931521 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff12c0668d0 0x7ff12c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:45.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.407+0000 7ff14d32f640 1 --2- 172.21.15.107:0/3871931521 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff12c0668d0 0x7ff12c068d90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff138001750 tx=0x7ff1380011a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:45.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.411+0000 7ff13e7fc640 1 -- 172.21.15.107:0/3871931521 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff140052380 con 0x7ff1340047d0 2023-12-04T05:44:45.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.599+0000 7ff14fd94640 1 -- 172.21.15.107:0/3871931521 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1480ccf60 con 0x7ff1340047d0 2023-12-04T05:44:45.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.599+0000 7ff13e7fc640 1 -- 172.21.15.107:0/3871931521 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5827 (secure 0 0 0) 0x7ff1400559a0 con 0x7ff1340047d0 2023-12-04T05:44:45.602 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:44:45.602 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:44:42.900817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4747,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:stopping","state_seq":45,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4747":{"gid":4747,"name":"b","rank":1,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.195:6836/2222928136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2222928136},{"type":"v1","addr":"172.21.15.195:6836","nonce":2222928136}]},"join_fscid":-1,"laggy_since":"2023-12-04T05:43:59.753818+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:44:45.603 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 32 2023-12-04T05:44:45.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.603+0000 7ff14fd94640 1 -- 172.21.15.107:0/3871931521 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff12c0668d0 msgr2=0x7ff12c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:45.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.603+0000 7ff14fd94640 1 --2- 172.21.15.107:0/3871931521 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff12c0668d0 0x7ff12c068d90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff138001750 tx=0x7ff1380011a0 comp rx=0 tx=0).stop 2023-12-04T05:44:45.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.603+0000 7ff14fd94640 1 -- 172.21.15.107:0/3871931521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1340047d0 msgr2=0x7ff14815a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:45.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.603+0000 7ff14fd94640 1 --2- 172.21.15.107:0/3871931521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1340047d0 0x7ff14815a3f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff140004700 tx=0x7ff140007700 comp rx=0 tx=0).stop 2023-12-04T05:44:45.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.603+0000 7ff14fd94640 1 -- 172.21.15.107:0/3871931521 shutdown_connections 2023-12-04T05:44:45.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.603+0000 7ff14fd94640 1 --2- 172.21.15.107:0/3871931521 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff12c0668d0 0x7ff12c068d90 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:45.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.603+0000 7ff14fd94640 1 --2- 172.21.15.107:0/3871931521 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff14806c080 0x7ff148140ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:45.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.603+0000 7ff14fd94640 1 --2- 172.21.15.107:0/3871931521 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff148151190 0x7ff14806ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:45.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.603+0000 7ff14fd94640 1 --2- 172.21.15.107:0/3871931521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1340047d0 0x7ff14815a3f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:45.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.603+0000 7ff14fd94640 1 -- 172.21.15.107:0/3871931521 >> 172.21.15.107:0/3871931521 conn(0x7ff14806e280 msgr2=0x7ff1480c4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:44:45.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.603+0000 7ff14fd94640 1 -- 172.21.15.107:0/3871931521 shutdown_connections 2023-12-04T05:44:45.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:45.603+0000 7ff14fd94640 1 -- 172.21.15.107:0/3871931521 wait complete. 2023-12-04T05:44:45.620 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:44:47.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:44:47.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.751+0000 7f0c643a8640 1 Processor -- start 2023-12-04T05:44:47.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.751+0000 7f0c643a8640 1 -- start start 2023-12-04T05:44:47.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.755+0000 7f0c643a8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0c5c072110 0x7f0c5c158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.755+0000 7f0c643a8640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0c5c0724f0 0x7f0c5c0c7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.755+0000 7f0c643a8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c5c158da0 con 0x7f0c5c0c56f0 2023-12-04T05:44:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.755+0000 7f0c643a8640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c5c1583d0 con 0x7f0c5c072110 2023-12-04T05:44:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.755+0000 7f0c643a8640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5c0c6170 con 0x7f0c5c0724f0 2023-12-04T05:44:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.755+0000 7f0c62144640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0c5c072110 0x7f0c5c158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.755+0000 7f0c62144640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0c5c072110 0x7f0c5c158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47614/0 (socket says 172.21.15.107:47614) 2023-12-04T05:44:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.755+0000 7f0c62144640 1 -- 172.21.15.107:0/1880232938 learned_addr learned my addr 172.21.15.107:0/1880232938 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:44:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.755+0000 7f0c62945640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0c5c0c56f0 0x7f0c5c140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58570/0 (socket says 172.21.15.107:58570) 2023-12-04T05:44:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.755+0000 7f0c61943640 1 --2- 172.21.15.107:0/1880232938 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0c5c0724f0 0x7f0c5c0c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.755+0000 7f0c62144640 1 -- 172.21.15.107:0/1880232938 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0c5c0724f0 msgr2=0x7f0c5c0c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.755+0000 7f0c62144640 1 --2- 172.21.15.107:0/1880232938 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0c5c0724f0 0x7f0c5c0c7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.755+0000 7f0c62144640 1 -- 172.21.15.107:0/1880232938 >> v1:172.21.15.107:6789/0 conn(0x7f0c5c0c56f0 legacy=0x7f0c5c140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.755+0000 7f0c62144640 1 -- 172.21.15.107:0/1880232938 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c5c157da0 con 0x7f0c5c072110 2023-12-04T05:44:47.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.755+0000 7f0c62144640 1 --2- 172.21.15.107:0/1880232938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c5c072110 0x7f0c5c158860 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0c54009870 tx=0x7f0c5401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b930392db535dc72 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:47.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.755+0000 7f0c61142640 1 -- 172.21.15.107:0/1880232938 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c540a8a30 con 0x7f0c5c072110 2023-12-04T05:44:47.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.755+0000 7f0c61142640 1 -- 172.21.15.107:0/1880232938 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c540a8b90 con 0x7f0c5c072110 2023-12-04T05:44:47.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.755+0000 7f0c61142640 1 -- 172.21.15.107:0/1880232938 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c540b0440 con 0x7f0c5c072110 2023-12-04T05:44:47.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c643a8640 1 -- 172.21.15.107:0/1880232938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c5c072110 msgr2=0x7f0c5c158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:47.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c643a8640 1 --2- 172.21.15.107:0/1880232938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c5c072110 0x7f0c5c158860 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0c54009870 tx=0x7f0c5401ed30 comp rx=0 tx=0).stop 2023-12-04T05:44:47.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c643a8640 1 -- 172.21.15.107:0/1880232938 shutdown_connections 2023-12-04T05:44:47.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c643a8640 1 --2- 172.21.15.107:0/1880232938 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0c5c0724f0 0x7f0c5c0c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:47.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c643a8640 1 --2- 172.21.15.107:0/1880232938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c5c072110 0x7f0c5c158860 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:47.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c643a8640 1 -- 172.21.15.107:0/1880232938 >> 172.21.15.107:0/1880232938 conn(0x7f0c5c0c7ff0 msgr2=0x7f0c5c0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:44:47.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c643a8640 1 -- 172.21.15.107:0/1880232938 shutdown_connections 2023-12-04T05:44:47.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c643a8640 1 -- 172.21.15.107:0/1880232938 wait complete. 2023-12-04T05:44:47.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c643a8640 1 Processor -- start 2023-12-04T05:44:47.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c643a8640 1 -- start start 2023-12-04T05:44:47.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c643a8640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0c5c072110 0x7f0c5c0ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:47.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c643a8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c5c0724f0 0x7f0c5c14b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:47.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c643a8640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c5c0c56f0 0x7f0c5c14bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:47.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c643a8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5c14a920 con 0x7f0c5c0724f0 2023-12-04T05:44:47.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c643a8640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5c14df40 con 0x7f0c5c0c56f0 2023-12-04T05:44:47.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c643a8640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5c14e200 con 0x7f0c5c072110 2023-12-04T05:44:47.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c61943640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c5c0724f0 0x7f0c5c14b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:47.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c61943640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c5c0724f0 0x7f0c5c14b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47626/0 (socket says 172.21.15.107:47626) 2023-12-04T05:44:47.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c61943640 1 -- 172.21.15.107:0/351325348 learned_addr learned my addr 172.21.15.107:0/351325348 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:44:47.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c61943640 1 -- 172.21.15.107:0/351325348 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0c5c072110 msgr2=0x7f0c5c0ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:47.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c61943640 1 --2- 172.21.15.107:0/351325348 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0c5c072110 0x7f0c5c0ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:47.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c61943640 1 -- 172.21.15.107:0/351325348 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c5c0c56f0 msgr2=0x7f0c5c14bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:47.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c62945640 1 --2- 172.21.15.107:0/351325348 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c5c0c56f0 0x7f0c5c14bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:47.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c61943640 1 --2- 172.21.15.107:0/351325348 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c5c0c56f0 0x7f0c5c14bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:47.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c61943640 1 -- 172.21.15.107:0/351325348 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c540094d0 con 0x7f0c5c0724f0 2023-12-04T05:44:47.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c61943640 1 --2- 172.21.15.107:0/351325348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c5c0724f0 0x7f0c5c14b770 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0c5000d800 tx=0x7f0c5000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:47.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c5affd640 1 -- 172.21.15.107:0/351325348 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5000b6c0 con 0x7f0c5c0724f0 2023-12-04T05:44:47.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c5affd640 1 -- 172.21.15.107:0/351325348 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5000b820 con 0x7f0c5c0724f0 2023-12-04T05:44:47.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c643a8640 1 -- 172.21.15.107:0/351325348 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c5c14e4e0 con 0x7f0c5c0724f0 2023-12-04T05:44:47.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c5affd640 1 -- 172.21.15.107:0/351325348 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5000bb90 con 0x7f0c5c0724f0 2023-12-04T05:44:47.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c643a8640 1 -- 172.21.15.107:0/351325348 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c5c14eaa0 con 0x7f0c5c0724f0 2023-12-04T05:44:47.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.759+0000 7f0c643a8640 1 -- 172.21.15.107:0/351325348 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c5c067870 con 0x7f0c5c0724f0 2023-12-04T05:44:47.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.763+0000 7f0c5affd640 1 -- 172.21.15.107:0/351325348 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f0c5000bcf0 con 0x7f0c5c0724f0 2023-12-04T05:44:47.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.763+0000 7f0c5affd640 1 --2- 172.21.15.107:0/351325348 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0c480668d0 0x7f0c48068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:47.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.763+0000 7f0c62144640 1 --2- 172.21.15.107:0/351325348 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0c480668d0 0x7f0c48068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:47.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.763+0000 7f0c5affd640 1 -- 172.21.15.107:0/351325348 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f0c50084af0 con 0x7f0c5c0724f0 2023-12-04T05:44:47.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.763+0000 7f0c62144640 1 --2- 172.21.15.107:0/351325348 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0c480668d0 0x7f0c48068d90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0c540b2000 tx=0x7f0c54004740 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:47.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.767+0000 7f0c5affd640 1 -- 172.21.15.107:0/351325348 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0c50051c10 con 0x7f0c5c0724f0 2023-12-04T05:44:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.955+0000 7f0c643a8640 1 -- 172.21.15.107:0/351325348 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c5c155750 con 0x7f0c5c0724f0 2023-12-04T05:44:47.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.955+0000 7f0c5affd640 1 -- 172.21.15.107:0/351325348 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5827 (secure 0 0 0) 0x7f0c50011340 con 0x7f0c5c0724f0 2023-12-04T05:44:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:44:47.960 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:44:42.900817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4747,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:stopping","state_seq":45,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4747":{"gid":4747,"name":"b","rank":1,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.195:6836/2222928136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2222928136},{"type":"v1","addr":"172.21.15.195:6836","nonce":2222928136}]},"join_fscid":-1,"laggy_since":"2023-12-04T05:43:59.753818+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:44:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 32 2023-12-04T05:44:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.959+0000 7f0c643a8640 1 -- 172.21.15.107:0/351325348 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0c480668d0 msgr2=0x7f0c48068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.959+0000 7f0c643a8640 1 --2- 172.21.15.107:0/351325348 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0c480668d0 0x7f0c48068d90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0c540b2000 tx=0x7f0c54004740 comp rx=0 tx=0).stop 2023-12-04T05:44:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.959+0000 7f0c643a8640 1 -- 172.21.15.107:0/351325348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c5c0724f0 msgr2=0x7f0c5c14b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.959+0000 7f0c643a8640 1 --2- 172.21.15.107:0/351325348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c5c0724f0 0x7f0c5c14b770 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0c5000d800 tx=0x7f0c5000dcc0 comp rx=0 tx=0).stop 2023-12-04T05:44:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.959+0000 7f0c643a8640 1 -- 172.21.15.107:0/351325348 shutdown_connections 2023-12-04T05:44:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.959+0000 7f0c643a8640 1 --2- 172.21.15.107:0/351325348 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0c480668d0 0x7f0c48068d90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:47.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.959+0000 7f0c643a8640 1 --2- 172.21.15.107:0/351325348 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c5c0c56f0 0x7f0c5c14bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:47.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.959+0000 7f0c643a8640 1 --2- 172.21.15.107:0/351325348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c5c0724f0 0x7f0c5c14b770 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:47.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.959+0000 7f0c643a8640 1 --2- 172.21.15.107:0/351325348 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0c5c072110 0x7f0c5c0ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:47.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.959+0000 7f0c643a8640 1 -- 172.21.15.107:0/351325348 >> 172.21.15.107:0/351325348 conn(0x7f0c5c0c7ff0 msgr2=0x7f0c5c156930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:44:47.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.959+0000 7f0c643a8640 1 -- 172.21.15.107:0/351325348 shutdown_connections 2023-12-04T05:44:47.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:47.959+0000 7f0c643a8640 1 -- 172.21.15.107:0/351325348 wait complete. 2023-12-04T05:44:47.976 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:44:48.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~60s 2023-12-04T05:44:48.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~101s 2023-12-04T05:44:49.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:44:50.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.103+0000 7f64f5b0c640 1 Processor -- start 2023-12-04T05:44:50.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.103+0000 7f64f5b0c640 1 -- start start 2023-12-04T05:44:50.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.107+0000 7f64f5b0c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f64f0154020 0x7f64f015a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:50.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.107+0000 7f64f5b0c640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f64f00d01c0 0x7f64f0071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:50.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.107+0000 7f64f5b0c640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f64f0072220 0x7f64f00c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:50.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.107+0000 7f64f5b0c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f00ca230 con 0x7f64f0154020 2023-12-04T05:44:50.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.107+0000 7f64f5b0c640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f00c6770 con 0x7f64f0072220 2023-12-04T05:44:50.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.107+0000 7f64f5b0c640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f015ad50 con 0x7f64f00d01c0 2023-12-04T05:44:50.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.107+0000 7f64ef7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f64f0154020 0x7f64f015a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:50.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.107+0000 7f64ef7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f64f0154020 0x7f64f015a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47634/0 (socket says 172.21.15.107:47634) 2023-12-04T05:44:50.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.107+0000 7f64ef7fe640 1 -- 172.21.15.107:0/3846199478 learned_addr learned my addr 172.21.15.107:0/3846199478 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:44:50.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.107+0000 7f64eeffd640 1 --2- 172.21.15.107:0/3846199478 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f64f00d01c0 0x7f64f0071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:50.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.107+0000 7f64effff640 1 --2- 172.21.15.107:0/3846199478 >> v2:172.21.15.195:3300/0 conn(0x7f64f0072220 0x7f64f00c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:50.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.107+0000 7f64ef7fe640 1 -- 172.21.15.107:0/3846199478 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f64f00d01c0 msgr2=0x7f64f0071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:50.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.107+0000 7f64ef7fe640 1 --2- 172.21.15.107:0/3846199478 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f64f00d01c0 0x7f64f0071ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:50.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.107+0000 7f64ef7fe640 1 -- 172.21.15.107:0/3846199478 >> v2:172.21.15.195:3300/0 conn(0x7f64f0072220 msgr2=0x7f64f00c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:50.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.107+0000 7f64ef7fe640 1 --2- 172.21.15.107:0/3846199478 >> v2:172.21.15.195:3300/0 conn(0x7f64f0072220 0x7f64f00c4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:50.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.107+0000 7f64ef7fe640 1 -- 172.21.15.107:0/3846199478 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f015d710 con 0x7f64f0154020 2023-12-04T05:44:50.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.107+0000 7f64ef7fe640 1 --2- 172.21.15.107:0/3846199478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64f0154020 0x7f64f015a810 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f64f00caa70 tx=0x7f64e001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22297b741a2ab1df server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:50.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64ee7fc640 1 -- 172.21.15.107:0/3846199478 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e00a89b0 con 0x7f64f0154020 2023-12-04T05:44:50.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64ee7fc640 1 -- 172.21.15.107:0/3846199478 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e00a8b10 con 0x7f64f0154020 2023-12-04T05:44:50.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64ee7fc640 1 -- 172.21.15.107:0/3846199478 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e00a8ea0 con 0x7f64f0154020 2023-12-04T05:44:50.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64f5b0c640 1 -- 172.21.15.107:0/3846199478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64f0154020 msgr2=0x7f64f015a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:50.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64f5b0c640 1 --2- 172.21.15.107:0/3846199478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64f0154020 0x7f64f015a810 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f64f00caa70 tx=0x7f64e001ecf0 comp rx=0 tx=0).stop 2023-12-04T05:44:50.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64f5b0c640 1 -- 172.21.15.107:0/3846199478 shutdown_connections 2023-12-04T05:44:50.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64f5b0c640 1 --2- 172.21.15.107:0/3846199478 >> v2:172.21.15.195:3300/0 conn(0x7f64f0072220 0x7f64f00c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:50.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64f5b0c640 1 --2- 172.21.15.107:0/3846199478 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f64f00d01c0 0x7f64f0071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:50.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64f5b0c640 1 --2- 172.21.15.107:0/3846199478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64f0154020 0x7f64f015a810 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:50.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64f5b0c640 1 -- 172.21.15.107:0/3846199478 >> 172.21.15.107:0/3846199478 conn(0x7f64f006c6c0 msgr2=0x7f64f00cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:44:50.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64f5b0c640 1 -- 172.21.15.107:0/3846199478 shutdown_connections 2023-12-04T05:44:50.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64f5b0c640 1 -- 172.21.15.107:0/3846199478 wait complete. 2023-12-04T05:44:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64f5b0c640 1 Processor -- start 2023-12-04T05:44:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64f5b0c640 1 -- start start 2023-12-04T05:44:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64f5b0c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64f0072220 0x7f64f00b2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64f5b0c640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64f00d01c0 0x7f64f014d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64f5b0c640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f64f0154020 0x7f64f01f1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64f5b0c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f014fc50 con 0x7f64f0072220 2023-12-04T05:44:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64f5b0c640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f0150810 con 0x7f64f00d01c0 2023-12-04T05:44:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64f5b0c640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f014f570 con 0x7f64f0154020 2023-12-04T05:44:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64ef7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64f0072220 0x7f64f00b2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64ef7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64f0072220 0x7f64f00b2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47642/0 (socket says 172.21.15.107:47642) 2023-12-04T05:44:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64ef7fe640 1 -- 172.21.15.107:0/1622887828 learned_addr learned my addr 172.21.15.107:0/1622887828 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:44:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64eeffd640 1 --2- 172.21.15.107:0/1622887828 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64f00d01c0 0x7f64f014d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64ef7fe640 1 -- 172.21.15.107:0/1622887828 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f64f0154020 msgr2=0x7f64f01f1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64ef7fe640 1 --2- 172.21.15.107:0/1622887828 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f64f0154020 0x7f64f01f1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64ef7fe640 1 -- 172.21.15.107:0/1622887828 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64f00d01c0 msgr2=0x7f64f014d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64ef7fe640 1 --2- 172.21.15.107:0/1622887828 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64f00d01c0 0x7f64f014d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64ef7fe640 1 -- 172.21.15.107:0/1622887828 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e00094d0 con 0x7f64f0072220 2023-12-04T05:44:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64ef7fe640 1 --2- 172.21.15.107:0/1622887828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64f0072220 0x7f64f00b2b80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f64e001d3a0 tx=0x7f64e001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64e7fff640 1 -- 172.21.15.107:0/1622887828 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e00ba600 con 0x7f64f0072220 2023-12-04T05:44:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64e7fff640 1 -- 172.21.15.107:0/1622887828 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e00ba780 con 0x7f64f0072220 2023-12-04T05:44:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64f5b0c640 1 -- 172.21.15.107:0/1622887828 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f00b3230 con 0x7f64f0072220 2023-12-04T05:44:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64e7fff640 1 -- 172.21.15.107:0/1622887828 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e00bab30 con 0x7f64f0072220 2023-12-04T05:44:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.111+0000 7f64f5b0c640 1 -- 172.21.15.107:0/1622887828 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f00b37f0 con 0x7f64f0072220 2023-12-04T05:44:50.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.115+0000 7f64f5b0c640 1 -- 172.21.15.107:0/1622887828 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f015a780 con 0x7f64f0072220 2023-12-04T05:44:50.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.115+0000 7f64e7fff640 1 -- 172.21.15.107:0/1622887828 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f64e00bac90 con 0x7f64f0072220 2023-12-04T05:44:50.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.115+0000 7f64e7fff640 1 --2- 172.21.15.107:0/1622887828 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f64d4066900 0x7f64d4068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:50.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.115+0000 7f64e7fff640 1 -- 172.21.15.107:0/1622887828 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f64e0122c40 con 0x7f64f0072220 2023-12-04T05:44:50.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.115+0000 7f64eeffd640 1 --2- 172.21.15.107:0/1622887828 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f64d4066900 0x7f64d4068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:50.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.115+0000 7f64eeffd640 1 --2- 172.21.15.107:0/1622887828 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f64d4066900 0x7f64d4068dc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f64f00b4870 tx=0x7f64e8008460 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:50.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.119+0000 7f64e7fff640 1 -- 172.21.15.107:0/1622887828 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f64e00efe80 con 0x7f64f0072220 2023-12-04T05:44:50.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.307+0000 7f64f5b0c640 1 -- 172.21.15.107:0/1622887828 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64f015a780 con 0x7f64f0072220 2023-12-04T05:44:50.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.307+0000 7f64e7fff640 1 -- 172.21.15.107:0/1622887828 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5827 (secure 0 0 0) 0x7f64e00b5090 con 0x7f64f0072220 2023-12-04T05:44:50.309 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:44:50.309 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:44:42.900817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4747,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:stopping","state_seq":45,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4747":{"gid":4747,"name":"b","rank":1,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.195:6836/2222928136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2222928136},{"type":"v1","addr":"172.21.15.195:6836","nonce":2222928136}]},"join_fscid":-1,"laggy_since":"2023-12-04T05:43:59.753818+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:44:50.309 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 32 2023-12-04T05:44:50.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.307+0000 7f64f5b0c640 1 -- 172.21.15.107:0/1622887828 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f64d4066900 msgr2=0x7f64d4068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:50.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.307+0000 7f64f5b0c640 1 --2- 172.21.15.107:0/1622887828 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f64d4066900 0x7f64d4068dc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f64f00b4870 tx=0x7f64e8008460 comp rx=0 tx=0).stop 2023-12-04T05:44:50.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.307+0000 7f64f5b0c640 1 -- 172.21.15.107:0/1622887828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64f0072220 msgr2=0x7f64f00b2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:50.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.307+0000 7f64f5b0c640 1 --2- 172.21.15.107:0/1622887828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64f0072220 0x7f64f00b2b80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f64e001d3a0 tx=0x7f64e001ba90 comp rx=0 tx=0).stop 2023-12-04T05:44:50.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.307+0000 7f64f5b0c640 1 -- 172.21.15.107:0/1622887828 shutdown_connections 2023-12-04T05:44:50.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.307+0000 7f64f5b0c640 1 --2- 172.21.15.107:0/1622887828 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f64d4066900 0x7f64d4068dc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:50.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.307+0000 7f64f5b0c640 1 --2- 172.21.15.107:0/1622887828 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f64f0154020 0x7f64f01f1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:50.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.307+0000 7f64f5b0c640 1 --2- 172.21.15.107:0/1622887828 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64f00d01c0 0x7f64f014d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:50.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.307+0000 7f64f5b0c640 1 --2- 172.21.15.107:0/1622887828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64f0072220 0x7f64f00b2b80 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:50.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.307+0000 7f64f5b0c640 1 -- 172.21.15.107:0/1622887828 >> 172.21.15.107:0/1622887828 conn(0x7f64f006c6c0 msgr2=0x7f64f015ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:44:50.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.311+0000 7f64f5b0c640 1 -- 172.21.15.107:0/1622887828 shutdown_connections 2023-12-04T05:44:50.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:50.311+0000 7f64f5b0c640 1 -- 172.21.15.107:0/1622887828 wait complete. 2023-12-04T05:44:50.328 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:44:52.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:44:52.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f4968b52640 1 Processor -- start 2023-12-04T05:44:52.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f4968b52640 1 -- start start 2023-12-04T05:44:52.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f4968b52640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4964151180 0x7f49640c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:52.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f4968b52640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4964070380 0x7f4964070790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:52.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f4968b52640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49640700c0 con 0x7f49640714f0 2023-12-04T05:44:52.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f4968b52640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4964156df0 con 0x7f4964070380 2023-12-04T05:44:52.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f4968b52640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49641570d0 con 0x7f4964151180 2023-12-04T05:44:52.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f4961d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f49640714f0 0x7f49640718f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58576/0 (socket says 172.21.15.107:58576) 2023-12-04T05:44:52.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f4961d9b640 1 -- 172.21.15.107:0/195511256 learned_addr learned my addr 172.21.15.107:0/195511256 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:44:52.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f4962d9d640 1 --2- 172.21.15.107:0/195511256 >> v2:172.21.15.195:3300/0 conn(0x7f4964070380 0x7f4964070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:52.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f496259c640 1 --2- 172.21.15.107:0/195511256 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4964151180 0x7f49640c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:52.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f496159a640 1 -- 172.21.15.107:0/195511256 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609577983 0 0) 0x7f49640700c0 con 0x7f49640714f0 2023-12-04T05:44:52.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f496159a640 1 -- 172.21.15.107:0/195511256 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f494c003530 con 0x7f49640714f0 2023-12-04T05:44:52.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f496159a640 1 -- 172.21.15.107:0/195511256 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 23204828 0 0) 0x7f494c003530 con 0x7f49640714f0 2023-12-04T05:44:52.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f496159a640 1 -- 172.21.15.107:0/195511256 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4964151180 msgr2=0x7f49640c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:52.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f496159a640 1 --2- 172.21.15.107:0/195511256 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4964151180 0x7f49640c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:52.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f496159a640 1 -- 172.21.15.107:0/195511256 >> v2:172.21.15.195:3300/0 conn(0x7f4964070380 msgr2=0x7f4964070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:52.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f496159a640 1 --2- 172.21.15.107:0/195511256 >> v2:172.21.15.195:3300/0 conn(0x7f4964070380 0x7f4964070790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:52.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f496159a640 1 -- 172.21.15.107:0/195511256 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49641573b0 con 0x7f49640714f0 2023-12-04T05:44:52.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f496159a640 1 -- 172.21.15.107:0/195511256 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f4958001680 con 0x7f49640714f0 2023-12-04T05:44:52.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f496159a640 1 -- 172.21.15.107:0/195511256 >> v1:172.21.15.107:6789/0 conn(0x7f49640714f0 legacy=0x7f49640718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:52.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f496159a640 1 --2- 172.21.15.107:0/195511256 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4964151180 0x7f494c003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:52.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f496159a640 1 --2- 172.21.15.107:0/195511256 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f494c003f00 0x7f494c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:52.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f496159a640 1 --2- 172.21.15.107:0/195511256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f494c004d10 0x7f494c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:52.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f496159a640 1 -- 172.21.15.107:0/195511256 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4964156df0 con 0x7f494c004d10 2023-12-04T05:44:52.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f496159a640 1 -- 172.21.15.107:0/195511256 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49641570d0 con 0x7f494c003f00 2023-12-04T05:44:52.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f496159a640 1 -- 172.21.15.107:0/195511256 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f494c00b990 con 0x7f4964151180 2023-12-04T05:44:52.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f4962d9d640 1 --2- 172.21.15.107:0/195511256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f494c004d10 0x7f494c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:52.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f4961d9b640 1 --2- 172.21.15.107:0/195511256 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f494c003f00 0x7f494c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:52.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f496259c640 1 --2- 172.21.15.107:0/195511256 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4964151180 0x7f494c003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:52.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f4962d9d640 1 -- 172.21.15.107:0/195511256 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4964151180 msgr2=0x7f494c003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:52.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f4962d9d640 1 --2- 172.21.15.107:0/195511256 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4964151180 0x7f494c003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:52.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f4962d9d640 1 -- 172.21.15.107:0/195511256 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f494c003f00 msgr2=0x7f494c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:52.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f4962d9d640 1 --2- 172.21.15.107:0/195511256 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f494c003f00 0x7f494c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:52.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.459+0000 7f4962d9d640 1 -- 172.21.15.107:0/195511256 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494c003530 con 0x7f494c004d10 2023-12-04T05:44:52.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.463+0000 7f4962d9d640 1 --2- 172.21.15.107:0/195511256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f494c004d10 0x7f494c005190 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f495c009fc0 tx=0x7f495c01cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5591288c03b6bad6 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:52.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.463+0000 7f496159a640 1 -- 172.21.15.107:0/195511256 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495c0a5cb0 con 0x7f494c004d10 2023-12-04T05:44:52.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.463+0000 7f496159a640 1 -- 172.21.15.107:0/195511256 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f495c00f040 con 0x7f494c004d10 2023-12-04T05:44:52.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.463+0000 7f496159a640 1 -- 172.21.15.107:0/195511256 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495c0ad680 con 0x7f494c004d10 2023-12-04T05:44:52.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.463+0000 7f4968b52640 1 -- 172.21.15.107:0/195511256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f494c004d10 msgr2=0x7f494c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:52.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.463+0000 7f4968b52640 1 --2- 172.21.15.107:0/195511256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f494c004d10 0x7f494c005190 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f495c009fc0 tx=0x7f495c01cc40 comp rx=0 tx=0).stop 2023-12-04T05:44:52.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.463+0000 7f4962d9d640 1 -- 172.21.15.107:0/195511256 reap_dead start 2023-12-04T05:44:52.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.463+0000 7f4968b52640 1 -- 172.21.15.107:0/195511256 shutdown_connections 2023-12-04T05:44:52.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.463+0000 7f4968b52640 1 -- 172.21.15.107:0/195511256 >> 172.21.15.107:0/195511256 conn(0x7f49640c7ff0 msgr2=0x7f49640c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:44:52.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.463+0000 7f4968b52640 1 -- 172.21.15.107:0/195511256 shutdown_connections 2023-12-04T05:44:52.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.463+0000 7f4968b52640 1 -- 172.21.15.107:0/195511256 wait complete. 2023-12-04T05:44:52.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.463+0000 7f4968b52640 1 Processor -- start 2023-12-04T05:44:52.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.463+0000 7f4968b52640 1 -- start start 2023-12-04T05:44:52.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4968b52640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f494c004d10 0x7f49641ee830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4968b52640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49640c4be0 0x7f4964143330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4968b52640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4964143870 0x7f4964141c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4968b52640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49640c5c50 con 0x7f49640c4be0 2023-12-04T05:44:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4968b52640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4964142f10 con 0x7f4964143870 2023-12-04T05:44:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4968b52640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49640c92f0 con 0x7f494c004d10 2023-12-04T05:44:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4961d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49640c4be0 0x7f4964143330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4961d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49640c4be0 0x7f4964143330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47658/0 (socket says 172.21.15.107:47658) 2023-12-04T05:44:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4961d9b640 1 -- 172.21.15.107:0/153383987 learned_addr learned my addr 172.21.15.107:0/153383987 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:44:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f496259c640 1 --2- 172.21.15.107:0/153383987 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f494c004d10 0x7f49641ee830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4962d9d640 1 --2- 172.21.15.107:0/153383987 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4964143870 0x7f4964141c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4961d9b640 1 -- 172.21.15.107:0/153383987 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f494c004d10 msgr2=0x7f49641ee830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4961d9b640 1 --2- 172.21.15.107:0/153383987 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f494c004d10 0x7f49641ee830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4961d9b640 1 -- 172.21.15.107:0/153383987 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4964143870 msgr2=0x7f4964141c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:52.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4961d9b640 1 --2- 172.21.15.107:0/153383987 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4964143870 0x7f4964141c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:52.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4961d9b640 1 -- 172.21.15.107:0/153383987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49640c5440 con 0x7f49640c4be0 2023-12-04T05:44:52.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4961d9b640 1 --2- 172.21.15.107:0/153383987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49640c4be0 0x7f4964143330 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f495800e880 tx=0x7f495800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:52.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4952ffd640 1 -- 172.21.15.107:0/153383987 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495800c8d0 con 0x7f49640c4be0 2023-12-04T05:44:52.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4952ffd640 1 -- 172.21.15.107:0/153383987 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f495800ca50 con 0x7f49640c4be0 2023-12-04T05:44:52.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4952ffd640 1 -- 172.21.15.107:0/153383987 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495800ce00 con 0x7f49640c4be0 2023-12-04T05:44:52.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4968b52640 1 -- 172.21.15.107:0/153383987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49640c5690 con 0x7f49640c4be0 2023-12-04T05:44:52.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4968b52640 1 -- 172.21.15.107:0/153383987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4964142850 con 0x7f49640c4be0 2023-12-04T05:44:52.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4968b52640 1 -- 172.21.15.107:0/153383987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4964143050 con 0x7f49640c4be0 2023-12-04T05:44:52.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4952ffd640 1 -- 172.21.15.107:0/153383987 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4958013420 con 0x7f49640c4be0 2023-12-04T05:44:52.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4952ffd640 1 --2- 172.21.15.107:0/153383987 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4944066900 0x7f4944068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:52.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f496259c640 1 --2- 172.21.15.107:0/153383987 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4944066900 0x7f4944068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:52.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f4952ffd640 1 -- 172.21.15.107:0/153383987 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f4958084320 con 0x7f49640c4be0 2023-12-04T05:44:52.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.467+0000 7f496259c640 1 --2- 172.21.15.107:0/153383987 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4944066900 0x7f4944068dc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4954009a30 tx=0x7f4954008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:52.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.471+0000 7f4952ffd640 1 -- 172.21.15.107:0/153383987 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4964143050 con 0x7f49640c4be0 2023-12-04T05:44:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.655+0000 7f4968b52640 1 -- 172.21.15.107:0/153383987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4964143050 con 0x7f49640c4be0 2023-12-04T05:44:52.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.655+0000 7f4952ffd640 1 -- 172.21.15.107:0/153383987 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5827 (secure 0 0 0) 0x7f4964143050 con 0x7f49640c4be0 2023-12-04T05:44:52.658 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:44:52.658 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:44:42.900817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4747,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:stopping","state_seq":45,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4747":{"gid":4747,"name":"b","rank":1,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.195:6836/2222928136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2222928136},{"type":"v1","addr":"172.21.15.195:6836","nonce":2222928136}]},"join_fscid":-1,"laggy_since":"2023-12-04T05:43:59.753818+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:44:52.662 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 32 2023-12-04T05:44:52.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.659+0000 7f4968b52640 1 -- 172.21.15.107:0/153383987 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4944066900 msgr2=0x7f4944068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:52.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.659+0000 7f4968b52640 1 --2- 172.21.15.107:0/153383987 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4944066900 0x7f4944068dc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4954009a30 tx=0x7f4954008040 comp rx=0 tx=0).stop 2023-12-04T05:44:52.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.659+0000 7f4968b52640 1 -- 172.21.15.107:0/153383987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49640c4be0 msgr2=0x7f4964143330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:52.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.659+0000 7f4968b52640 1 --2- 172.21.15.107:0/153383987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49640c4be0 0x7f4964143330 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f495800e880 tx=0x7f495800ed40 comp rx=0 tx=0).stop 2023-12-04T05:44:52.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.659+0000 7f4968b52640 1 -- 172.21.15.107:0/153383987 shutdown_connections 2023-12-04T05:44:52.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.659+0000 7f4968b52640 1 --2- 172.21.15.107:0/153383987 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4944066900 0x7f4944068dc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:52.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.659+0000 7f4968b52640 1 --2- 172.21.15.107:0/153383987 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4964143870 0x7f4964141c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:52.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.659+0000 7f4968b52640 1 --2- 172.21.15.107:0/153383987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49640c4be0 0x7f4964143330 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:52.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.659+0000 7f4968b52640 1 --2- 172.21.15.107:0/153383987 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f494c004d10 0x7f49641ee830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:52.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.659+0000 7f4968b52640 1 -- 172.21.15.107:0/153383987 >> 172.21.15.107:0/153383987 conn(0x7f49640c7ff0 msgr2=0x7f49640c70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:44:52.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.659+0000 7f4968b52640 1 -- 172.21.15.107:0/153383987 shutdown_connections 2023-12-04T05:44:52.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:52.659+0000 7f4968b52640 1 -- 172.21.15.107:0/153383987 wait complete. 2023-12-04T05:44:52.675 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:44:54.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:44:54.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.807+0000 7f5ec8880640 1 Processor -- start 2023-12-04T05:44:54.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.807+0000 7f5ec8880640 1 -- start start 2023-12-04T05:44:54.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec8880640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5ec4071c00 0x7f5ec40c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:54.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec8880640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5ec4070380 0x7f5ec40707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:54.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec8880640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec40700c0 con 0x7f5ec4151180 2023-12-04T05:44:54.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec8880640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ec4156df0 con 0x7f5ec4071c00 2023-12-04T05:44:54.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec8880640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec41570d0 con 0x7f5ec4070380 2023-12-04T05:44:54.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5ec4151180 0x7f5ec40714f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58582/0 (socket says 172.21.15.107:58582) 2023-12-04T05:44:54.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec259c640 1 -- 172.21.15.107:0/1255673768 learned_addr learned my addr 172.21.15.107:0/1255673768 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:44:54.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec2d9d640 1 --2- 172.21.15.107:0/1255673768 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5ec4070380 0x7f5ec40707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:54.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec1d9b640 1 --2- 172.21.15.107:0/1255673768 >> v2:172.21.15.195:3300/0 conn(0x7f5ec4071c00 0x7f5ec40c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:54.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec159a640 1 -- 172.21.15.107:0/1255673768 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302401699 0 0) 0x7f5ec40700c0 con 0x7f5ec4151180 2023-12-04T05:44:54.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec159a640 1 -- 172.21.15.107:0/1255673768 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eac003530 con 0x7f5ec4151180 2023-12-04T05:44:54.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec159a640 1 -- 172.21.15.107:0/1255673768 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2738295418 0 0) 0x7f5eac003530 con 0x7f5ec4151180 2023-12-04T05:44:54.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec159a640 1 -- 172.21.15.107:0/1255673768 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5ec4070380 msgr2=0x7f5ec40707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:54.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec159a640 1 --2- 172.21.15.107:0/1255673768 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5ec4070380 0x7f5ec40707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:54.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec159a640 1 -- 172.21.15.107:0/1255673768 >> v2:172.21.15.195:3300/0 conn(0x7f5ec4071c00 msgr2=0x7f5ec40c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:54.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec159a640 1 --2- 172.21.15.107:0/1255673768 >> v2:172.21.15.195:3300/0 conn(0x7f5ec4071c00 0x7f5ec40c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:54.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec159a640 1 -- 172.21.15.107:0/1255673768 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec41573b0 con 0x7f5ec4151180 2023-12-04T05:44:54.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec159a640 1 -- 172.21.15.107:0/1255673768 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f5eb40016a0 con 0x7f5ec4151180 2023-12-04T05:44:54.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec159a640 1 -- 172.21.15.107:0/1255673768 >> v1:172.21.15.107:6789/0 conn(0x7f5ec4151180 legacy=0x7f5ec40714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:54.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec159a640 1 --2- 172.21.15.107:0/1255673768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5eac003980 0x7f5eac003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:54.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec159a640 1 --2- 172.21.15.107:0/1255673768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5eac0047d0 0x7f5eac004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:54.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec159a640 1 --2- 172.21.15.107:0/1255673768 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5ec4070380 0x7f5eac005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:54.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec159a640 1 -- 172.21.15.107:0/1255673768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec4156df0 con 0x7f5eac003980 2023-12-04T05:44:54.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec159a640 1 -- 172.21.15.107:0/1255673768 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec41570d0 con 0x7f5eac0047d0 2023-12-04T05:44:54.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec159a640 1 -- 172.21.15.107:0/1255673768 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eac00b990 con 0x7f5ec4070380 2023-12-04T05:44:54.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec259c640 1 --2- 172.21.15.107:0/1255673768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5eac003980 0x7f5eac003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:54.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec1d9b640 1 --2- 172.21.15.107:0/1255673768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5eac0047d0 0x7f5eac004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:54.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec2d9d640 1 --2- 172.21.15.107:0/1255673768 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5ec4070380 0x7f5eac005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:54.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec259c640 1 -- 172.21.15.107:0/1255673768 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5ec4070380 msgr2=0x7f5eac005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:54.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec259c640 1 --2- 172.21.15.107:0/1255673768 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5ec4070380 0x7f5eac005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:54.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec259c640 1 -- 172.21.15.107:0/1255673768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5eac0047d0 msgr2=0x7f5eac004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:54.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec259c640 1 --2- 172.21.15.107:0/1255673768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5eac0047d0 0x7f5eac004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:54.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.811+0000 7f5ec259c640 1 -- 172.21.15.107:0/1255673768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eac003530 con 0x7f5eac003980 2023-12-04T05:44:54.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec259c640 1 --2- 172.21.15.107:0/1255673768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5eac003980 0x7f5eac003da0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5eb400c290 tx=0x7f5eb401f430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6481d75a7ebd3f00 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:54.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec159a640 1 -- 172.21.15.107:0/1255673768 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb40058e0 con 0x7f5eac003980 2023-12-04T05:44:54.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec159a640 1 -- 172.21.15.107:0/1255673768 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb4005a40 con 0x7f5eac003980 2023-12-04T05:44:54.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec159a640 1 -- 172.21.15.107:0/1255673768 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb4005df0 con 0x7f5eac003980 2023-12-04T05:44:54.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec8880640 1 -- 172.21.15.107:0/1255673768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5eac003980 msgr2=0x7f5eac003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:54.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec8880640 1 --2- 172.21.15.107:0/1255673768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5eac003980 0x7f5eac003da0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5eb400c290 tx=0x7f5eb401f430 comp rx=0 tx=0).stop 2023-12-04T05:44:54.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec2d9d640 1 -- 172.21.15.107:0/1255673768 reap_dead start 2023-12-04T05:44:54.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec8880640 1 -- 172.21.15.107:0/1255673768 shutdown_connections 2023-12-04T05:44:54.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec8880640 1 -- 172.21.15.107:0/1255673768 >> 172.21.15.107:0/1255673768 conn(0x7f5ec40c7ff0 msgr2=0x7f5ec40c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:44:54.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec8880640 1 -- 172.21.15.107:0/1255673768 shutdown_connections 2023-12-04T05:44:54.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec8880640 1 -- 172.21.15.107:0/1255673768 wait complete. 2023-12-04T05:44:54.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec8880640 1 Processor -- start 2023-12-04T05:44:54.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec8880640 1 -- start start 2023-12-04T05:44:54.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec8880640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5eac003980 0x7f5ec40c47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:54.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec8880640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ec414c8b0 0x7f5ec414b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:54.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec8880640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ec414bbd0 0x7f5ec41eeac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:54.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec8880640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec4154ce0 con 0x7f5ec414c8b0 2023-12-04T05:44:54.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec8880640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec414c060 con 0x7f5ec414bbd0 2023-12-04T05:44:54.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec8880640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec414c340 con 0x7f5eac003980 2023-12-04T05:44:54.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ec414c8b0 0x7f5ec414b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:54.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ec414c8b0 0x7f5ec414b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47680/0 (socket says 172.21.15.107:47680) 2023-12-04T05:44:54.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec1d9b640 1 -- 172.21.15.107:0/3714787783 learned_addr learned my addr 172.21.15.107:0/3714787783 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:44:54.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec2d9d640 1 --2- 172.21.15.107:0/3714787783 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ec414bbd0 0x7f5ec41eeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:54.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.815+0000 7f5ec259c640 1 --2- 172.21.15.107:0/3714787783 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5eac003980 0x7f5ec40c47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:54.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.819+0000 7f5ec1d9b640 1 -- 172.21.15.107:0/3714787783 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5eac003980 msgr2=0x7f5ec40c47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:54.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.819+0000 7f5ec1d9b640 1 --2- 172.21.15.107:0/3714787783 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5eac003980 0x7f5ec40c47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:54.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.819+0000 7f5ec1d9b640 1 -- 172.21.15.107:0/3714787783 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ec414bbd0 msgr2=0x7f5ec41eeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:54.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.819+0000 7f5ec1d9b640 1 --2- 172.21.15.107:0/3714787783 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ec414bbd0 0x7f5ec41eeac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:54.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.819+0000 7f5ec1d9b640 1 -- 172.21.15.107:0/3714787783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec414c620 con 0x7f5ec414c8b0 2023-12-04T05:44:54.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.819+0000 7f5ec259c640 1 --2- 172.21.15.107:0/3714787783 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5eac003980 0x7f5ec40c47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:44:54.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.819+0000 7f5ec1d9b640 1 --2- 172.21.15.107:0/3714787783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ec414c8b0 0x7f5ec414b690 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5eb80115e0 tx=0x7f5eb8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:54.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.819+0000 7f5eb2ffd640 1 -- 172.21.15.107:0/3714787783 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb800f8f0 con 0x7f5ec414c8b0 2023-12-04T05:44:54.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.819+0000 7f5eb2ffd640 1 -- 172.21.15.107:0/3714787783 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb800fa50 con 0x7f5ec414c8b0 2023-12-04T05:44:54.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.819+0000 7f5ec8880640 1 -- 172.21.15.107:0/3714787783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec40c3fc0 con 0x7f5ec414c8b0 2023-12-04T05:44:54.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.819+0000 7f5eb2ffd640 1 -- 172.21.15.107:0/3714787783 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb800fdc0 con 0x7f5ec414c8b0 2023-12-04T05:44:54.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.819+0000 7f5ec8880640 1 -- 172.21.15.107:0/3714787783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec40c4d00 con 0x7f5ec414c8b0 2023-12-04T05:44:54.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.819+0000 7f5ec8880640 1 -- 172.21.15.107:0/3714787783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec4067870 con 0x7f5ec414c8b0 2023-12-04T05:44:54.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.819+0000 7f5eb2ffd640 1 -- 172.21.15.107:0/3714787783 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f5eb8015460 con 0x7f5ec414c8b0 2023-12-04T05:44:54.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.819+0000 7f5eb2ffd640 1 --2- 172.21.15.107:0/3714787783 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5ea40668f0 0x7f5ea4068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:54.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.819+0000 7f5eb2ffd640 1 -- 172.21.15.107:0/3714787783 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f5eb8084a30 con 0x7f5ec414c8b0 2023-12-04T05:44:54.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.819+0000 7f5ec259c640 1 --2- 172.21.15.107:0/3714787783 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5ea40668f0 0x7f5ea4068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:54.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.819+0000 7f5ec259c640 1 --2- 172.21.15.107:0/3714787783 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5ea40668f0 0x7f5ea4068db0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5eb40ae000 tx=0x7f5eb40aa070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:54.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:54.823+0000 7f5eb2ffd640 1 -- 172.21.15.107:0/3714787783 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5eb8051ac0 con 0x7f5ec414c8b0 2023-12-04T05:44:54.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~67s 2023-12-04T05:44:54.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~108s 2023-12-04T05:44:55.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:55.011+0000 7f5ec8880640 1 -- 172.21.15.107:0/3714787783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ec40c2210 con 0x7f5ec414c8b0 2023-12-04T05:44:55.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:55.015+0000 7f5eb2ffd640 1 -- 172.21.15.107:0/3714787783 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5827 (secure 0 0 0) 0x7f5eb8022020 con 0x7f5ec414c8b0 2023-12-04T05:44:55.017 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:44:55.017 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:44:42.900817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4747,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:stopping","state_seq":45,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4747":{"gid":4747,"name":"b","rank":1,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.195:6836/2222928136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2222928136},{"type":"v1","addr":"172.21.15.195:6836","nonce":2222928136}]},"join_fscid":-1,"laggy_since":"2023-12-04T05:43:59.753818+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:44:55.020 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 32 2023-12-04T05:44:55.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:55.015+0000 7f5ec8880640 1 -- 172.21.15.107:0/3714787783 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5ea40668f0 msgr2=0x7f5ea4068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:55.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:55.015+0000 7f5ec8880640 1 --2- 172.21.15.107:0/3714787783 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5ea40668f0 0x7f5ea4068db0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5eb40ae000 tx=0x7f5eb40aa070 comp rx=0 tx=0).stop 2023-12-04T05:44:55.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:55.015+0000 7f5ec8880640 1 -- 172.21.15.107:0/3714787783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ec414c8b0 msgr2=0x7f5ec414b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:55.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:55.015+0000 7f5ec8880640 1 --2- 172.21.15.107:0/3714787783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ec414c8b0 0x7f5ec414b690 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5eb80115e0 tx=0x7f5eb8011aa0 comp rx=0 tx=0).stop 2023-12-04T05:44:55.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:55.015+0000 7f5ec8880640 1 -- 172.21.15.107:0/3714787783 shutdown_connections 2023-12-04T05:44:55.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:55.015+0000 7f5ec8880640 1 --2- 172.21.15.107:0/3714787783 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5ea40668f0 0x7f5ea4068db0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:55.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:55.015+0000 7f5ec8880640 1 --2- 172.21.15.107:0/3714787783 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ec414bbd0 0x7f5ec41eeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:55.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:55.015+0000 7f5ec8880640 1 --2- 172.21.15.107:0/3714787783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ec414c8b0 0x7f5ec414b690 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:55.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:55.015+0000 7f5ec8880640 1 --2- 172.21.15.107:0/3714787783 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5eac003980 0x7f5ec40c47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:55.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:55.015+0000 7f5ec8880640 1 -- 172.21.15.107:0/3714787783 >> 172.21.15.107:0/3714787783 conn(0x7f5ec40c7ff0 msgr2=0x7f5ec406a5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:44:55.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:55.015+0000 7f5ec8880640 1 -- 172.21.15.107:0/3714787783 shutdown_connections 2023-12-04T05:44:55.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:55.015+0000 7f5ec8880640 1 -- 172.21.15.107:0/3714787783 wait complete. 2023-12-04T05:44:55.035 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-12-04T05:44:55.036 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 32, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-12-04T05:39:59.032019+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 32, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4247": { "addr": "172.21.15.107:6839/993693883", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6838", "nonce": 993693883, "type": "v2" }, { "addr": "172.21.15.107:6839", "nonce": 993693883, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 18, "join_fscid": -1, "name": "a", "rank": 4, "state": "up:stopping", "state_seq": 45 }, "gid_4260": { "addr": "172.21.15.107:6836/3444851515", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 3444851515, "type": "v2" }, { "addr": "172.21.15.107:6836", "nonce": 3444851515, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4260, "incarnation": 14, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 9 }, "gid_4263": { "addr": "172.21.15.107:6837/2210527429", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6835", "nonce": 2210527429, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 2210527429, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4263, "incarnation": 12, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 7 }, "gid_4288": { "addr": "172.21.15.195:6837/3412865477", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6835", "nonce": 3412865477, "type": "v2" }, { "addr": "172.21.15.195:6837", "nonce": 3412865477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4747": { "addr": "172.21.15.195:6836/2222928136", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2222928136, "type": "v2" }, { "addr": "172.21.15.195:6836", "nonce": 2222928136, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4747, "incarnation": 25, "join_fscid": -1, "laggy_since": "2023-12-04T05:43:59.753818+0000", "name": "b", "rank": 1, "state": "up:active", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 25, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-12-04T05:44:42.900817+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 4747, "mds_2": 4263, "mds_3": 4260, "mds_4": 4247 } } } ], "standbys": [] } 2023-12-04T05:44:57.038 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:44:57.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89d09d8640 1 Processor -- start 2023-12-04T05:44:57.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89d09d8640 1 -- start start 2023-12-04T05:44:57.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89d09d8640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f89cc071c00 0x7f89cc0c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:57.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89d09d8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89cc0700c0 con 0x7f89cc151180 2023-12-04T05:44:57.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89d09d8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89cc156df0 con 0x7f89cc070380 2023-12-04T05:44:57.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89d09d8640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc1570d0 con 0x7f89cc071c00 2023-12-04T05:44:57.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89ca59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f89cc151180 0x7f89cc0714f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58592/0 (socket says 172.21.15.107:58592) 2023-12-04T05:44:57.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89ca59c640 1 -- 172.21.15.107:0/1576384231 learned_addr learned my addr 172.21.15.107:0/1576384231 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:44:57.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89c3fff640 1 --2- 172.21.15.107:0/1576384231 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f89cc071c00 0x7f89cc0c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:57.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89c9d9b640 1 -- 172.21.15.107:0/1576384231 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712294054 0 0) 0x7f89cc0700c0 con 0x7f89cc151180 2023-12-04T05:44:57.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89c9d9b640 1 -- 172.21.15.107:0/1576384231 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89b8005740 con 0x7f89cc151180 2023-12-04T05:44:57.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89c9d9b640 1 -- 172.21.15.107:0/1576384231 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3587562833 0 0) 0x7f89b8005740 con 0x7f89cc151180 2023-12-04T05:44:57.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89c9d9b640 1 -- 172.21.15.107:0/1576384231 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f89cc071c00 msgr2=0x7f89cc0c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:57.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89c9d9b640 1 --2- 172.21.15.107:0/1576384231 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f89cc071c00 0x7f89cc0c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:57.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89c9d9b640 1 -- 172.21.15.107:0/1576384231 >> v1:172.21.15.195:6789/0 conn(0x7f89cc070380 legacy=0x7f89cc070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:57.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89c9d9b640 1 -- 172.21.15.107:0/1576384231 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89cc1573b0 con 0x7f89cc151180 2023-12-04T05:44:57.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89c9d9b640 1 -- 172.21.15.107:0/1576384231 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f89bc001680 con 0x7f89cc151180 2023-12-04T05:44:57.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89c9d9b640 1 -- 172.21.15.107:0/1576384231 >> v1:172.21.15.107:6789/0 conn(0x7f89cc151180 legacy=0x7f89cc0714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:57.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89c9d9b640 1 --2- 172.21.15.107:0/1576384231 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f89cc071c00 0x7f89b8005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:57.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89c9d9b640 1 --2- 172.21.15.107:0/1576384231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89b8006140 0x7f89b80069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:57.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89c9d9b640 1 --2- 172.21.15.107:0/1576384231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89b8006f20 0x7f89b80073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:57.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89c9d9b640 1 -- 172.21.15.107:0/1576384231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc1570d0 con 0x7f89b8006140 2023-12-04T05:44:57.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89c9d9b640 1 -- 172.21.15.107:0/1576384231 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b800dc80 con 0x7f89b8006f20 2023-12-04T05:44:57.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89c9d9b640 1 -- 172.21.15.107:0/1576384231 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b800de40 con 0x7f89cc071c00 2023-12-04T05:44:57.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89c3fff640 1 --2- 172.21.15.107:0/1576384231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89b8006140 0x7f89b80069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:57.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89ca59c640 1 --2- 172.21.15.107:0/1576384231 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f89cc071c00 0x7f89b8005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:57.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89cad9d640 1 --2- 172.21.15.107:0/1576384231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89b8006f20 0x7f89b80073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:57.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89c3fff640 1 -- 172.21.15.107:0/1576384231 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f89cc071c00 msgr2=0x7f89b8005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:57.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89c3fff640 1 --2- 172.21.15.107:0/1576384231 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f89cc071c00 0x7f89b8005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:57.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89c3fff640 1 -- 172.21.15.107:0/1576384231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89b8006f20 msgr2=0x7f89b80073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:57.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89c3fff640 1 --2- 172.21.15.107:0/1576384231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89b8006f20 0x7f89b80073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:57.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.167+0000 7f89c3fff640 1 -- 172.21.15.107:0/1576384231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b8005740 con 0x7f89b8006140 2023-12-04T05:44:57.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.171+0000 7f89c3fff640 1 --2- 172.21.15.107:0/1576384231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89b8006140 0x7f89b80069e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f89b4009ae0 tx=0x7f89b401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d12dc4a2e3215b server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:57.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.171+0000 7f89c9d9b640 1 -- 172.21.15.107:0/1576384231 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b40a5530 con 0x7f89b8006140 2023-12-04T05:44:57.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.171+0000 7f89c9d9b640 1 -- 172.21.15.107:0/1576384231 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b40a5690 con 0x7f89b8006140 2023-12-04T05:44:57.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.171+0000 7f89c9d9b640 1 -- 172.21.15.107:0/1576384231 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b40a5a40 con 0x7f89b8006140 2023-12-04T05:44:57.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.171+0000 7f89d09d8640 1 -- 172.21.15.107:0/1576384231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89b8006140 msgr2=0x7f89b80069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:57.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.171+0000 7f89d09d8640 1 --2- 172.21.15.107:0/1576384231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89b8006140 0x7f89b80069e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f89b4009ae0 tx=0x7f89b401ca50 comp rx=0 tx=0).stop 2023-12-04T05:44:57.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.171+0000 7f89cad9d640 1 -- 172.21.15.107:0/1576384231 reap_dead start 2023-12-04T05:44:57.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.171+0000 7f89d09d8640 1 -- 172.21.15.107:0/1576384231 shutdown_connections 2023-12-04T05:44:57.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.171+0000 7f89d09d8640 1 -- 172.21.15.107:0/1576384231 >> 172.21.15.107:0/1576384231 conn(0x7f89cc0c7ff0 msgr2=0x7f89cc0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:44:57.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.171+0000 7f89d09d8640 1 -- 172.21.15.107:0/1576384231 shutdown_connections 2023-12-04T05:44:57.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.171+0000 7f89d09d8640 1 -- 172.21.15.107:0/1576384231 wait complete. 2023-12-04T05:44:57.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.171+0000 7f89d09d8640 1 Processor -- start 2023-12-04T05:44:57.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.171+0000 7f89d09d8640 1 -- start start 2023-12-04T05:44:57.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.171+0000 7f89d09d8640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f89b8006140 0x7f89cc153640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:57.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.171+0000 7f89d09d8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89cc1541c0 0x7f89cc0c98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:57.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.171+0000 7f89d09d8640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89cc14b7d0 0x7f89cc14bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:57.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.171+0000 7f89d09d8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc153b80 con 0x7f89cc1541c0 2023-12-04T05:44:57.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.171+0000 7f89d09d8640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc152cd0 con 0x7f89cc14b7d0 2023-12-04T05:44:57.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.171+0000 7f89d09d8640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc153f00 con 0x7f89b8006140 2023-12-04T05:44:57.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.171+0000 7f89c3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89cc1541c0 0x7f89cc0c98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:57.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.171+0000 7f89c3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89cc1541c0 0x7f89cc0c98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47692/0 (socket says 172.21.15.107:47692) 2023-12-04T05:44:57.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.171+0000 7f89c3fff640 1 -- 172.21.15.107:0/3127307741 learned_addr learned my addr 172.21.15.107:0/3127307741 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:44:57.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.175+0000 7f89ca59c640 1 --2- 172.21.15.107:0/3127307741 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f89b8006140 0x7f89cc153640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:57.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.175+0000 7f89cad9d640 1 --2- 172.21.15.107:0/3127307741 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89cc14b7d0 0x7f89cc14bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:57.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.175+0000 7f89c3fff640 1 -- 172.21.15.107:0/3127307741 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f89b8006140 msgr2=0x7f89cc153640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:57.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.175+0000 7f89c3fff640 1 --2- 172.21.15.107:0/3127307741 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f89b8006140 0x7f89cc153640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:57.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.175+0000 7f89c3fff640 1 -- 172.21.15.107:0/3127307741 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89cc14b7d0 msgr2=0x7f89cc14bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:57.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.175+0000 7f89c3fff640 1 --2- 172.21.15.107:0/3127307741 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89cc14b7d0 0x7f89cc14bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:57.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.175+0000 7f89c3fff640 1 -- 172.21.15.107:0/3127307741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89cc0ca540 con 0x7f89cc1541c0 2023-12-04T05:44:57.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.175+0000 7f89cad9d640 1 --2- 172.21.15.107:0/3127307741 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89cc14b7d0 0x7f89cc14bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:44:57.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.175+0000 7f89c3fff640 1 --2- 172.21.15.107:0/3127307741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89cc1541c0 0x7f89cc0c98f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f89b4009ae0 tx=0x7f89b8007a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:57.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.175+0000 7f89c2ffd640 1 -- 172.21.15.107:0/3127307741 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b4004220 con 0x7f89cc1541c0 2023-12-04T05:44:57.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.175+0000 7f89c2ffd640 1 -- 172.21.15.107:0/3127307741 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b4004380 con 0x7f89cc1541c0 2023-12-04T05:44:57.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.175+0000 7f89c2ffd640 1 -- 172.21.15.107:0/3127307741 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b40ad850 con 0x7f89cc1541c0 2023-12-04T05:44:57.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.175+0000 7f89d09d8640 1 -- 172.21.15.107:0/3127307741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89cc152100 con 0x7f89cc1541c0 2023-12-04T05:44:57.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.175+0000 7f89d09d8640 1 -- 172.21.15.107:0/3127307741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89cc1523a0 con 0x7f89cc1541c0 2023-12-04T05:44:57.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.175+0000 7f89d09d8640 1 -- 172.21.15.107:0/3127307741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89cc067870 con 0x7f89cc1541c0 2023-12-04T05:44:57.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.175+0000 7f89c2ffd640 1 -- 172.21.15.107:0/3127307741 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f89b40b4040 con 0x7f89cc1541c0 2023-12-04T05:44:57.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.175+0000 7f89c2ffd640 1 --2- 172.21.15.107:0/3127307741 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f89ac066900 0x7f89ac068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:57.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.175+0000 7f89ca59c640 1 --2- 172.21.15.107:0/3127307741 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f89ac066900 0x7f89ac068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:57.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.175+0000 7f89c2ffd640 1 -- 172.21.15.107:0/3127307741 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f89b411e980 con 0x7f89cc1541c0 2023-12-04T05:44:57.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.175+0000 7f89ca59c640 1 --2- 172.21.15.107:0/3127307741 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f89ac066900 0x7f89ac068dc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f89bc0053a0 tx=0x7f89bc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:57.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.179+0000 7f89c2ffd640 1 -- 172.21.15.107:0/3127307741 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f89b40ebd00 con 0x7f89cc1541c0 2023-12-04T05:44:57.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.359+0000 7f89d09d8640 1 -- 172.21.15.107:0/3127307741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89cc0c26d0 con 0x7f89cc1541c0 2023-12-04T05:44:57.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.359+0000 7f89c2ffd640 1 -- 172.21.15.107:0/3127307741 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5827 (secure 0 0 0) 0x7f89b40ef320 con 0x7f89cc1541c0 2023-12-04T05:44:57.362 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:44:57.363 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:44:42.900817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4747,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:stopping","state_seq":45,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4747":{"gid":4747,"name":"b","rank":1,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.195:6836/2222928136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2222928136},{"type":"v1","addr":"172.21.15.195:6836","nonce":2222928136}]},"join_fscid":-1,"laggy_since":"2023-12-04T05:43:59.753818+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:44:57.363 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 32 2023-12-04T05:44:57.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.359+0000 7f89d09d8640 1 -- 172.21.15.107:0/3127307741 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f89ac066900 msgr2=0x7f89ac068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:57.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.359+0000 7f89d09d8640 1 --2- 172.21.15.107:0/3127307741 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f89ac066900 0x7f89ac068dc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f89bc0053a0 tx=0x7f89bc009040 comp rx=0 tx=0).stop 2023-12-04T05:44:57.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.359+0000 7f89d09d8640 1 -- 172.21.15.107:0/3127307741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89cc1541c0 msgr2=0x7f89cc0c98f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:57.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.359+0000 7f89d09d8640 1 --2- 172.21.15.107:0/3127307741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89cc1541c0 0x7f89cc0c98f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f89b4009ae0 tx=0x7f89b8007a10 comp rx=0 tx=0).stop 2023-12-04T05:44:57.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.359+0000 7f89d09d8640 1 -- 172.21.15.107:0/3127307741 shutdown_connections 2023-12-04T05:44:57.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.363+0000 7f89d09d8640 1 --2- 172.21.15.107:0/3127307741 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f89ac066900 0x7f89ac068dc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:57.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.363+0000 7f89d09d8640 1 --2- 172.21.15.107:0/3127307741 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f89cc14b7d0 0x7f89cc14bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:57.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.363+0000 7f89d09d8640 1 --2- 172.21.15.107:0/3127307741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89cc1541c0 0x7f89cc0c98f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:57.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.363+0000 7f89d09d8640 1 --2- 172.21.15.107:0/3127307741 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f89b8006140 0x7f89cc153640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:57.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.363+0000 7f89d09d8640 1 -- 172.21.15.107:0/3127307741 >> 172.21.15.107:0/3127307741 conn(0x7f89cc0c7ff0 msgr2=0x7f89cc0c74a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:44:57.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.363+0000 7f89d09d8640 1 -- 172.21.15.107:0/3127307741 shutdown_connections 2023-12-04T05:44:57.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:57.363+0000 7f89d09d8640 1 -- 172.21.15.107:0/3127307741 wait complete. 2023-12-04T05:44:57.381 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:44:59.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:44:59.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.511+0000 7f466d218640 1 Processor -- start 2023-12-04T05:44:59.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.515+0000 7f466d218640 1 -- start start 2023-12-04T05:44:59.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.515+0000 7f466d218640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4668150550 0x7f466806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:59.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.515+0000 7f466d218640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f46680c6f70 0x7f466806c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:59.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.515+0000 7f466d218640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4668072460 0x7f46680728f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:59.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.515+0000 7f466d218640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46680c7ee0 con 0x7f4668150550 2023-12-04T05:44:59.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.515+0000 7f466d218640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46680c87a0 con 0x7f46680c6f70 2023-12-04T05:44:59.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.515+0000 7f466d218640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4668150930 con 0x7f4668072460 2023-12-04T05:44:59.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.515+0000 7f4666d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4668150550 0x7f466806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:59.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.515+0000 7f4666d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4668150550 0x7f466806bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54658/0 (socket says 172.21.15.107:54658) 2023-12-04T05:44:59.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.515+0000 7f4666d9d640 1 -- 172.21.15.107:0/289155537 learned_addr learned my addr 172.21.15.107:0/289155537 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:44:59.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.515+0000 7f466659c640 1 --2- 172.21.15.107:0/289155537 >> v2:172.21.15.195:3300/0 conn(0x7f46680c6f70 0x7f466806c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:59.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.515+0000 7f466759e640 1 --2- 172.21.15.107:0/289155537 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4668072460 0x7f46680728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:59.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.515+0000 7f4666d9d640 1 -- 172.21.15.107:0/289155537 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4668072460 msgr2=0x7f46680728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:59.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.515+0000 7f4666d9d640 1 --2- 172.21.15.107:0/289155537 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4668072460 0x7f46680728f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:59.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.515+0000 7f4666d9d640 1 -- 172.21.15.107:0/289155537 >> v2:172.21.15.195:3300/0 conn(0x7f46680c6f70 msgr2=0x7f466806c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:44:59.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.515+0000 7f4666d9d640 1 --2- 172.21.15.107:0/289155537 >> v2:172.21.15.195:3300/0 conn(0x7f46680c6f70 0x7f466806c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:59.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.515+0000 7f4666d9d640 1 -- 172.21.15.107:0/289155537 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46680c7c20 con 0x7f4668150550 2023-12-04T05:44:59.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.515+0000 7f466659c640 1 --2- 172.21.15.107:0/289155537 >> v2:172.21.15.195:3300/0 conn(0x7f46680c6f70 0x7f466806c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:44:59.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.515+0000 7f4666d9d640 1 --2- 172.21.15.107:0/289155537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4668150550 0x7f466806bca0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f466806d970 tx=0x7f465801c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43df7a8c7a509e55 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:59.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f4665d9b640 1 -- 172.21.15.107:0/289155537 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46580a57f0 con 0x7f4668150550 2023-12-04T05:44:59.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f4665d9b640 1 -- 172.21.15.107:0/289155537 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46580a5950 con 0x7f4668150550 2023-12-04T05:44:59.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f4665d9b640 1 -- 172.21.15.107:0/289155537 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46580a5ce0 con 0x7f4668150550 2023-12-04T05:44:59.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466d218640 1 -- 172.21.15.107:0/289155537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4668150550 msgr2=0x7f466806bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:59.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466d218640 1 --2- 172.21.15.107:0/289155537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4668150550 0x7f466806bca0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f466806d970 tx=0x7f465801c9d0 comp rx=0 tx=0).stop 2023-12-04T05:44:59.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466d218640 1 -- 172.21.15.107:0/289155537 shutdown_connections 2023-12-04T05:44:59.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466d218640 1 --2- 172.21.15.107:0/289155537 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4668072460 0x7f46680728f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:59.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466d218640 1 --2- 172.21.15.107:0/289155537 >> v2:172.21.15.195:3300/0 conn(0x7f46680c6f70 0x7f466806c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:59.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466d218640 1 --2- 172.21.15.107:0/289155537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4668150550 0x7f466806bca0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:59.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466d218640 1 -- 172.21.15.107:0/289155537 >> 172.21.15.107:0/289155537 conn(0x7f46680cbef0 msgr2=0x7f46680cc300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:44:59.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466d218640 1 -- 172.21.15.107:0/289155537 shutdown_connections 2023-12-04T05:44:59.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466d218640 1 -- 172.21.15.107:0/289155537 wait complete. 2023-12-04T05:44:59.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466d218640 1 Processor -- start 2023-12-04T05:44:59.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466d218640 1 -- start start 2023-12-04T05:44:59.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466d218640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4668072460 0x7f46681f16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:59.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466d218640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46680c6f70 0x7f4668149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:59.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466d218640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4668150550 0x7f466814eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:59.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466d218640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466814e690 con 0x7f46680c6f70 2023-12-04T05:44:59.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466d218640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466814f940 con 0x7f4668072460 2023-12-04T05:44:59.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466d218640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466814e040 con 0x7f4668150550 2023-12-04T05:44:59.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46680c6f70 0x7f4668149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:59.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46680c6f70 0x7f4668149940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54674/0 (socket says 172.21.15.107:54674) 2023-12-04T05:44:59.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466659c640 1 -- 172.21.15.107:0/4077888813 learned_addr learned my addr 172.21.15.107:0/4077888813 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:44:59.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466759e640 1 --2- 172.21.15.107:0/4077888813 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4668150550 0x7f466814eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:59.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f4666d9d640 1 --2- 172.21.15.107:0/4077888813 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4668072460 0x7f46681f16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:59.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466659c640 1 -- 172.21.15.107:0/4077888813 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4668150550 msgr2=0x7f466814eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:59.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466659c640 1 --2- 172.21.15.107:0/4077888813 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4668150550 0x7f466814eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:59.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466659c640 1 -- 172.21.15.107:0/4077888813 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4668072460 msgr2=0x7f46681f16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:59.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466659c640 1 --2- 172.21.15.107:0/4077888813 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4668072460 0x7f46681f16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:59.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466659c640 1 -- 172.21.15.107:0/4077888813 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4658009b30 con 0x7f46680c6f70 2023-12-04T05:44:59.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466659c640 1 --2- 172.21.15.107:0/4077888813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46680c6f70 0x7f4668149940 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f465c00c7f0 tx=0x7f465c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:59.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f464f7fe640 1 -- 172.21.15.107:0/4077888813 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c005ba0 con 0x7f46680c6f70 2023-12-04T05:44:59.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f464f7fe640 1 -- 172.21.15.107:0/4077888813 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f465c005d00 con 0x7f46680c6f70 2023-12-04T05:44:59.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466d218640 1 -- 172.21.15.107:0/4077888813 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466814e320 con 0x7f46680c6f70 2023-12-04T05:44:59.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f464f7fe640 1 -- 172.21.15.107:0/4077888813 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c007a50 con 0x7f46680c6f70 2023-12-04T05:44:59.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.519+0000 7f466d218640 1 -- 172.21.15.107:0/4077888813 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f466814fc20 con 0x7f46680c6f70 2023-12-04T05:44:59.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.523+0000 7f466d218640 1 -- 172.21.15.107:0/4077888813 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f466814e800 con 0x7f46680c6f70 2023-12-04T05:44:59.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.523+0000 7f464f7fe640 1 -- 172.21.15.107:0/4077888813 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f465c007bb0 con 0x7f46680c6f70 2023-12-04T05:44:59.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.523+0000 7f464f7fe640 1 --2- 172.21.15.107:0/4077888813 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f46540668d0 0x7f4654068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:44:59.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.523+0000 7f464f7fe640 1 -- 172.21.15.107:0/4077888813 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f465c084470 con 0x7f46680c6f70 2023-12-04T05:44:59.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.523+0000 7f4666d9d640 1 --2- 172.21.15.107:0/4077888813 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f46540668d0 0x7f4654068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:44:59.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.523+0000 7f4666d9d640 1 --2- 172.21.15.107:0/4077888813 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f46540668d0 0x7f4654068d90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f46580ae000 tx=0x7f46580a7070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:44:59.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.527+0000 7f464f7fe640 1 -- 172.21.15.107:0/4077888813 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f465c051620 con 0x7f46680c6f70 2023-12-04T05:44:59.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.719+0000 7f466d218640 1 -- 172.21.15.107:0/4077888813 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4668067ec0 con 0x7f46680c6f70 2023-12-04T05:44:59.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.719+0000 7f464f7fe640 1 -- 172.21.15.107:0/4077888813 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5827 (secure 0 0 0) 0x7f465c05b020 con 0x7f46680c6f70 2023-12-04T05:44:59.721 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:44:59.722 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:44:42.900817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4747,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:stopping","state_seq":45,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4747":{"gid":4747,"name":"b","rank":1,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.195:6836/2222928136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2222928136},{"type":"v1","addr":"172.21.15.195:6836","nonce":2222928136}]},"join_fscid":-1,"laggy_since":"2023-12-04T05:43:59.753818+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:44:59.722 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 32 2023-12-04T05:44:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.723+0000 7f466d218640 1 -- 172.21.15.107:0/4077888813 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f46540668d0 msgr2=0x7f4654068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.723+0000 7f466d218640 1 --2- 172.21.15.107:0/4077888813 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f46540668d0 0x7f4654068d90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f46580ae000 tx=0x7f46580a7070 comp rx=0 tx=0).stop 2023-12-04T05:44:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.723+0000 7f466d218640 1 -- 172.21.15.107:0/4077888813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46680c6f70 msgr2=0x7f4668149940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:44:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.723+0000 7f466d218640 1 --2- 172.21.15.107:0/4077888813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46680c6f70 0x7f4668149940 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f465c00c7f0 tx=0x7f465c00cbf0 comp rx=0 tx=0).stop 2023-12-04T05:44:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.723+0000 7f466d218640 1 -- 172.21.15.107:0/4077888813 shutdown_connections 2023-12-04T05:44:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.723+0000 7f466d218640 1 --2- 172.21.15.107:0/4077888813 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f46540668d0 0x7f4654068d90 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.723+0000 7f466d218640 1 --2- 172.21.15.107:0/4077888813 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4668150550 0x7f466814eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.723+0000 7f466d218640 1 --2- 172.21.15.107:0/4077888813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46680c6f70 0x7f4668149940 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:59.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.723+0000 7f466d218640 1 --2- 172.21.15.107:0/4077888813 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4668072460 0x7f46681f16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:44:59.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.723+0000 7f466d218640 1 -- 172.21.15.107:0/4077888813 >> 172.21.15.107:0/4077888813 conn(0x7f46680cbef0 msgr2=0x7f46680cb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:44:59.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.723+0000 7f466d218640 1 -- 172.21.15.107:0/4077888813 shutdown_connections 2023-12-04T05:44:59.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:44:59.723+0000 7f466d218640 1 -- 172.21.15.107:0/4077888813 wait complete. 2023-12-04T05:44:59.741 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:45:00.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:45:00.573 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:45:01.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~74s 2023-12-04T05:45:01.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~115s 2023-12-04T05:45:01.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:01.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.871+0000 7f933e338640 1 Processor -- start 2023-12-04T05:45:01.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.871+0000 7f933e338640 1 -- start start 2023-12-04T05:45:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.871+0000 7f933e338640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9338072110 0x7f9338158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.871+0000 7f933e338640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f93380724f0 0x7f93380c7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.871+0000 7f933e338640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9338158da0 con 0x7f93380c56f0 2023-12-04T05:45:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.871+0000 7f933e338640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93381583d0 con 0x7f9338072110 2023-12-04T05:45:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.871+0000 7f933e338640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93380c6170 con 0x7f93380724f0 2023-12-04T05:45:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.871+0000 7f933c8d5640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f93380c56f0 0x7f9338140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33886/0 (socket says 172.21.15.107:33886) 2023-12-04T05:45:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.871+0000 7f933c8d5640 1 -- 172.21.15.107:0/3452991190 learned_addr learned my addr 172.21.15.107:0/3452991190 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:45:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.871+0000 7f9337fff640 1 --2- 172.21.15.107:0/3452991190 >> v2:172.21.15.195:3300/0 conn(0x7f9338072110 0x7f9338158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.871+0000 7f93377fe640 1 --2- 172.21.15.107:0/3452991190 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f93380724f0 0x7f93380c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.871+0000 7f9336ffd640 1 -- 172.21.15.107:0/3452991190 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2843757622 0 0) 0x7f93300014a0 con 0x7f93380c56f0 2023-12-04T05:45:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.871+0000 7f9336ffd640 1 -- 172.21.15.107:0/3452991190 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9320003530 con 0x7f93380c56f0 2023-12-04T05:45:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f9336ffd640 1 -- 172.21.15.107:0/3452991190 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2656075542 0 0) 0x7f9320003530 con 0x7f93380c56f0 2023-12-04T05:45:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f9336ffd640 1 -- 172.21.15.107:0/3452991190 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f93380724f0 msgr2=0x7f93380c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f9336ffd640 1 --2- 172.21.15.107:0/3452991190 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f93380724f0 0x7f93380c7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f9336ffd640 1 -- 172.21.15.107:0/3452991190 >> v2:172.21.15.195:3300/0 conn(0x7f9338072110 msgr2=0x7f9338158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f9336ffd640 1 --2- 172.21.15.107:0/3452991190 >> v2:172.21.15.195:3300/0 conn(0x7f9338072110 0x7f9338158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f9336ffd640 1 -- 172.21.15.107:0/3452991190 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9338157da0 con 0x7f93380c56f0 2023-12-04T05:45:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f9336ffd640 1 -- 172.21.15.107:0/3452991190 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f9330001b90 con 0x7f93380c56f0 2023-12-04T05:45:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f9336ffd640 1 -- 172.21.15.107:0/3452991190 >> v1:172.21.15.107:6789/0 conn(0x7f93380c56f0 legacy=0x7f9338140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f9336ffd640 1 --2- 172.21.15.107:0/3452991190 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9320003980 0x7f9320003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f9336ffd640 1 --2- 172.21.15.107:0/3452991190 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f93380724f0 0x7f93200047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f9336ffd640 1 --2- 172.21.15.107:0/3452991190 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9320004d10 0x7f9320005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f9336ffd640 1 -- 172.21.15.107:0/3452991190 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93381583d0 con 0x7f9320004d10 2023-12-04T05:45:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f9336ffd640 1 -- 172.21.15.107:0/3452991190 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93380c6170 con 0x7f9320003980 2023-12-04T05:45:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f9336ffd640 1 -- 172.21.15.107:0/3452991190 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f932000ba10 con 0x7f93380724f0 2023-12-04T05:45:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f933c8d5640 1 --2- 172.21.15.107:0/3452991190 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9320004d10 0x7f9320005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f93377fe640 1 --2- 172.21.15.107:0/3452991190 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f93380724f0 0x7f93200047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f9337fff640 1 --2- 172.21.15.107:0/3452991190 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9320003980 0x7f9320003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f933c8d5640 1 -- 172.21.15.107:0/3452991190 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f93380724f0 msgr2=0x7f93200047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f933c8d5640 1 --2- 172.21.15.107:0/3452991190 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f93380724f0 0x7f93200047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f933c8d5640 1 -- 172.21.15.107:0/3452991190 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9320003980 msgr2=0x7f9320003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f933c8d5640 1 --2- 172.21.15.107:0/3452991190 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9320003980 0x7f9320003da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f933c8d5640 1 -- 172.21.15.107:0/3452991190 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9320003530 con 0x7f9320004d10 2023-12-04T05:45:01.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f933c8d5640 1 --2- 172.21.15.107:0/3452991190 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9320004d10 0x7f9320005190 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f933000a780 tx=0x7f933001d280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14bea64f8e371869 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:01.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f9336ffd640 1 -- 172.21.15.107:0/3452991190 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93300ad630 con 0x7f9320004d10 2023-12-04T05:45:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f9336ffd640 1 -- 172.21.15.107:0/3452991190 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93300a8070 con 0x7f9320004d10 2023-12-04T05:45:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f9336ffd640 1 -- 172.21.15.107:0/3452991190 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93300ad9f0 con 0x7f9320004d10 2023-12-04T05:45:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f933e338640 1 -- 172.21.15.107:0/3452991190 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9320004d10 msgr2=0x7f9320005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f933e338640 1 --2- 172.21.15.107:0/3452991190 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9320004d10 0x7f9320005190 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f933000a780 tx=0x7f933001d280 comp rx=0 tx=0).stop 2023-12-04T05:45:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f933c8d5640 1 -- 172.21.15.107:0/3452991190 reap_dead start 2023-12-04T05:45:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f933e338640 1 -- 172.21.15.107:0/3452991190 shutdown_connections 2023-12-04T05:45:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f933e338640 1 -- 172.21.15.107:0/3452991190 >> 172.21.15.107:0/3452991190 conn(0x7f93380c7ff0 msgr2=0x7f93380c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f933e338640 1 -- 172.21.15.107:0/3452991190 shutdown_connections 2023-12-04T05:45:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.875+0000 7f933e338640 1 -- 172.21.15.107:0/3452991190 wait complete. 2023-12-04T05:45:01.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f933e338640 1 Processor -- start 2023-12-04T05:45:01.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f933e338640 1 -- start start 2023-12-04T05:45:01.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f933e338640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9320004d10 0x7f933815a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:01.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f933e338640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f933815b510 0x7f933815b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:01.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f933e338640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9338070f50 0x7f93380713d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:01.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f933e338640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933815a0f0 con 0x7f9338070f50 2023-12-04T05:45:01.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f933e338640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93381f2b00 con 0x7f933815b510 2023-12-04T05:45:01.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f933e338640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9338071910 con 0x7f9320004d10 2023-12-04T05:45:01.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f933c8d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9338070f50 0x7f93380713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:01.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f933c8d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9338070f50 0x7f93380713d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54690/0 (socket says 172.21.15.107:54690) 2023-12-04T05:45:01.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f933c8d5640 1 -- 172.21.15.107:0/2840007103 learned_addr learned my addr 172.21.15.107:0/2840007103 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:01.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f9337fff640 1 --2- 172.21.15.107:0/2840007103 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9320004d10 0x7f933815a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:01.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f933c8d5640 1 -- 172.21.15.107:0/2840007103 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9320004d10 msgr2=0x7f933815a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:01.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f93377fe640 1 --2- 172.21.15.107:0/2840007103 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f933815b510 0x7f933815b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:01.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f933c8d5640 1 --2- 172.21.15.107:0/2840007103 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9320004d10 0x7f933815a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:01.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f933c8d5640 1 -- 172.21.15.107:0/2840007103 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f933815b510 msgr2=0x7f933815b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:01.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f933c8d5640 1 --2- 172.21.15.107:0/2840007103 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f933815b510 0x7f933815b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:01.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f933c8d5640 1 -- 172.21.15.107:0/2840007103 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93381594b0 con 0x7f9338070f50 2023-12-04T05:45:01.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f933c8d5640 1 --2- 172.21.15.107:0/2840007103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9338070f50 0x7f93380713d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f933001d0b0 tx=0x7f9320005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:01.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f9334ff9640 1 -- 172.21.15.107:0/2840007103 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9330001950 con 0x7f9338070f50 2023-12-04T05:45:01.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f9334ff9640 1 -- 172.21.15.107:0/2840007103 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9330001ab0 con 0x7f9338070f50 2023-12-04T05:45:01.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f933e338640 1 -- 172.21.15.107:0/2840007103 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9338155710 con 0x7f9338070f50 2023-12-04T05:45:01.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f9334ff9640 1 -- 172.21.15.107:0/2840007103 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9330004d70 con 0x7f9338070f50 2023-12-04T05:45:01.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.879+0000 7f933e338640 1 -- 172.21.15.107:0/2840007103 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9338159a30 con 0x7f9338070f50 2023-12-04T05:45:01.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.883+0000 7f933e338640 1 -- 172.21.15.107:0/2840007103 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f933815a230 con 0x7f9338070f50 2023-12-04T05:45:01.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.883+0000 7f9334ff9640 1 -- 172.21.15.107:0/2840007103 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f93300aa030 con 0x7f9338070f50 2023-12-04T05:45:01.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.883+0000 7f9334ff9640 1 --2- 172.21.15.107:0/2840007103 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f93240668d0 0x7f9324068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:01.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.883+0000 7f9334ff9640 1 -- 172.21.15.107:0/2840007103 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f933011e0a0 con 0x7f9338070f50 2023-12-04T05:45:01.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.883+0000 7f9337fff640 1 --2- 172.21.15.107:0/2840007103 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f93240668d0 0x7f9324068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:01.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.883+0000 7f9337fff640 1 --2- 172.21.15.107:0/2840007103 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f93240668d0 0x7f9324068d90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9328009910 tx=0x7f9328008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:01.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:01.887+0000 7f9334ff9640 1 -- 172.21.15.107:0/2840007103 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f933815a230 con 0x7f9338070f50 2023-12-04T05:45:02.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:02.075+0000 7f933e338640 1 -- 172.21.15.107:0/2840007103 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9338158da0 con 0x7f9338070f50 2023-12-04T05:45:02.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:02.075+0000 7f9334ff9640 1 -- 172.21.15.107:0/2840007103 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5827 (secure 0 0 0) 0x7f9338158da0 con 0x7f9338070f50 2023-12-04T05:45:02.076 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:02.077 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:44:42.900817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4747,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:stopping","state_seq":45,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4747":{"gid":4747,"name":"b","rank":1,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.195:6836/2222928136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2222928136},{"type":"v1","addr":"172.21.15.195:6836","nonce":2222928136}]},"join_fscid":-1,"laggy_since":"2023-12-04T05:43:59.753818+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:02.077 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 32 2023-12-04T05:45:02.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:02.075+0000 7f933e338640 1 -- 172.21.15.107:0/2840007103 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f93240668d0 msgr2=0x7f9324068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:02.075+0000 7f933e338640 1 --2- 172.21.15.107:0/2840007103 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f93240668d0 0x7f9324068d90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9328009910 tx=0x7f9328008040 comp rx=0 tx=0).stop 2023-12-04T05:45:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:02.075+0000 7f933e338640 1 -- 172.21.15.107:0/2840007103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9338070f50 msgr2=0x7f93380713d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:02.075+0000 7f933e338640 1 --2- 172.21.15.107:0/2840007103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9338070f50 0x7f93380713d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f933001d0b0 tx=0x7f9320005800 comp rx=0 tx=0).stop 2023-12-04T05:45:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:02.075+0000 7f933e338640 1 -- 172.21.15.107:0/2840007103 shutdown_connections 2023-12-04T05:45:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:02.075+0000 7f933e338640 1 --2- 172.21.15.107:0/2840007103 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f93240668d0 0x7f9324068d90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:02.075+0000 7f933e338640 1 --2- 172.21.15.107:0/2840007103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9338070f50 0x7f93380713d0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:02.075+0000 7f933e338640 1 --2- 172.21.15.107:0/2840007103 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f933815b510 0x7f933815b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:02.075+0000 7f933e338640 1 --2- 172.21.15.107:0/2840007103 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9320004d10 0x7f933815a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:02.075+0000 7f933e338640 1 -- 172.21.15.107:0/2840007103 >> 172.21.15.107:0/2840007103 conn(0x7f93380c7ff0 msgr2=0x7f9338155c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:02.075+0000 7f933e338640 1 -- 172.21.15.107:0/2840007103 shutdown_connections 2023-12-04T05:45:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:02.075+0000 7f933e338640 1 -- 172.21.15.107:0/2840007103 wait complete. 2023-12-04T05:45:02.094 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:45:04.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:04.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.223+0000 7f5a9241e640 1 Processor -- start 2023-12-04T05:45:04.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.223+0000 7f5a9241e640 1 -- start start 2023-12-04T05:45:04.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.227+0000 7f5a9241e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5a8c0ca2d0 0x7f5a8c072570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:04.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.227+0000 7f5a9241e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a8c072ab0 0x7f5a8c151960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:04.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.227+0000 7f5a9241e640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8c0c9570 con 0x7f5a8c0ca6b0 2023-12-04T05:45:04.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.227+0000 7f5a9241e640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a8c073290 con 0x7f5a8c0ca2d0 2023-12-04T05:45:04.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.227+0000 7f5a9241e640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c0c8c80 con 0x7f5a8c072ab0 2023-12-04T05:45:04.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.227+0000 7f5a909bb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a8c072ab0 0x7f5a8c151960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:04.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.227+0000 7f5a909bb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a8c072ab0 0x7f5a8c151960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.107:40802/0 (socket says 172.21.15.107:40802) 2023-12-04T05:45:04.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.227+0000 7f5a909bb640 1 -- 172.21.15.107:0/1560221172 learned_addr learned my addr 172.21.15.107:0/1560221172 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:04.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.227+0000 7f5a8bfff640 1 --2- 172.21.15.107:0/1560221172 >> v2:172.21.15.195:3300/0 conn(0x7f5a8c0ca2d0 0x7f5a8c072570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:04.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.227+0000 7f5a909bb640 1 -- 172.21.15.107:0/1560221172 >> v2:172.21.15.195:3300/0 conn(0x7f5a8c0ca2d0 msgr2=0x7f5a8c072570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:04.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.227+0000 7f5a909bb640 1 --2- 172.21.15.107:0/1560221172 >> v2:172.21.15.195:3300/0 conn(0x7f5a8c0ca2d0 0x7f5a8c072570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:04.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.227+0000 7f5a909bb640 1 -- 172.21.15.107:0/1560221172 >> v1:172.21.15.195:6789/0 conn(0x7f5a8c0ca6b0 legacy=0x7f5a8c0cd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:04.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.227+0000 7f5a909bb640 1 -- 172.21.15.107:0/1560221172 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8c072fb0 con 0x7f5a8c072ab0 2023-12-04T05:45:04.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a909bb640 1 --2- 172.21.15.107:0/1560221172 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a8c072ab0 0x7f5a8c151960 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5a84009e80 tx=0x7f5a8401f1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21a668e9e1cd480 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:04.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a8affd640 1 -- 172.21.15.107:0/1560221172 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a840af400 con 0x7f5a8c072ab0 2023-12-04T05:45:04.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a8affd640 1 -- 172.21.15.107:0/1560221172 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a840ab050 con 0x7f5a8c072ab0 2023-12-04T05:45:04.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a8affd640 1 -- 172.21.15.107:0/1560221172 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a840af780 con 0x7f5a8c072ab0 2023-12-04T05:45:04.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a9241e640 1 -- 172.21.15.107:0/1560221172 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a8c072ab0 msgr2=0x7f5a8c151960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:04.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a9241e640 1 --2- 172.21.15.107:0/1560221172 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a8c072ab0 0x7f5a8c151960 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5a84009e80 tx=0x7f5a8401f1e0 comp rx=0 tx=0).stop 2023-12-04T05:45:04.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a9241e640 1 -- 172.21.15.107:0/1560221172 shutdown_connections 2023-12-04T05:45:04.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a9241e640 1 --2- 172.21.15.107:0/1560221172 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a8c072ab0 0x7f5a8c151960 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:04.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a9241e640 1 --2- 172.21.15.107:0/1560221172 >> v2:172.21.15.195:3300/0 conn(0x7f5a8c0ca2d0 0x7f5a8c072570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:04.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a9241e640 1 -- 172.21.15.107:0/1560221172 >> 172.21.15.107:0/1560221172 conn(0x7f5a8c0663a0 msgr2=0x7f5a8c0667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:04.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a9241e640 1 -- 172.21.15.107:0/1560221172 shutdown_connections 2023-12-04T05:45:04.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a9241e640 1 -- 172.21.15.107:0/1560221172 wait complete. 2023-12-04T05:45:04.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a9241e640 1 Processor -- start 2023-12-04T05:45:04.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a9241e640 1 -- start start 2023-12-04T05:45:04.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a9241e640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a8c072ab0 0x7f5a8c146fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:04.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a9241e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a8c0ca2d0 0x7f5a8c1efa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:04.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a9241e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a8c14f1c0 0x7f5a8c15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:04.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a9241e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c1f3dd0 con 0x7f5a8c0ca2d0 2023-12-04T05:45:04.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a9241e640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c06cfc0 con 0x7f5a8c072ab0 2023-12-04T05:45:04.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a9241e640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c1f3010 con 0x7f5a8c14f1c0 2023-12-04T05:45:04.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a8b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a8c0ca2d0 0x7f5a8c1efa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:04.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a8b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a8c0ca2d0 0x7f5a8c1efa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54692/0 (socket says 172.21.15.107:54692) 2023-12-04T05:45:04.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a8b7fe640 1 -- 172.21.15.107:0/1784430976 learned_addr learned my addr 172.21.15.107:0/1784430976 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:04.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a909bb640 1 --2- 172.21.15.107:0/1784430976 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a8c14f1c0 0x7f5a8c15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:04.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a8bfff640 1 --2- 172.21.15.107:0/1784430976 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a8c072ab0 0x7f5a8c146fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:04.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a8b7fe640 1 -- 172.21.15.107:0/1784430976 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a8c14f1c0 msgr2=0x7f5a8c15b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:04.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a8b7fe640 1 --2- 172.21.15.107:0/1784430976 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a8c14f1c0 0x7f5a8c15b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:04.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a8b7fe640 1 -- 172.21.15.107:0/1784430976 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a8c072ab0 msgr2=0x7f5a8c146fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:04.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a8b7fe640 1 --2- 172.21.15.107:0/1784430976 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a8c072ab0 0x7f5a8c146fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:04.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a8b7fe640 1 -- 172.21.15.107:0/1784430976 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a84009b30 con 0x7f5a8c0ca2d0 2023-12-04T05:45:04.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.231+0000 7f5a8b7fe640 1 --2- 172.21.15.107:0/1784430976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a8c0ca2d0 0x7f5a8c1efa00 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5a7400c340 tx=0x7f5a7400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:04.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.235+0000 7f5a88ff9640 1 -- 172.21.15.107:0/1784430976 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a740052d0 con 0x7f5a8c0ca2d0 2023-12-04T05:45:04.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.235+0000 7f5a88ff9640 1 -- 172.21.15.107:0/1784430976 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a74005430 con 0x7f5a8c0ca2d0 2023-12-04T05:45:04.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.235+0000 7f5a9241e640 1 -- 172.21.15.107:0/1784430976 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a8c1f32f0 con 0x7f5a8c0ca2d0 2023-12-04T05:45:04.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.235+0000 7f5a88ff9640 1 -- 172.21.15.107:0/1784430976 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a74007260 con 0x7f5a8c0ca2d0 2023-12-04T05:45:04.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.235+0000 7f5a9241e640 1 -- 172.21.15.107:0/1784430976 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a8c1f38b0 con 0x7f5a8c0ca2d0 2023-12-04T05:45:04.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.235+0000 7f5a9241e640 1 -- 172.21.15.107:0/1784430976 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a8c067ec0 con 0x7f5a8c0ca2d0 2023-12-04T05:45:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.235+0000 7f5a88ff9640 1 -- 172.21.15.107:0/1784430976 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f5a740073c0 con 0x7f5a8c0ca2d0 2023-12-04T05:45:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.235+0000 7f5a88ff9640 1 --2- 172.21.15.107:0/1784430976 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5a6c0668d0 0x7f5a6c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.235+0000 7f5a88ff9640 1 -- 172.21.15.107:0/1784430976 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f5a74083860 con 0x7f5a8c0ca2d0 2023-12-04T05:45:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.235+0000 7f5a8bfff640 1 --2- 172.21.15.107:0/1784430976 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5a6c0668d0 0x7f5a6c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.235+0000 7f5a8bfff640 1 --2- 172.21.15.107:0/1784430976 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5a6c0668d0 0x7f5a6c068d90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c009af0 tx=0x7f5a7c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:04.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.239+0000 7f5a88ff9640 1 -- 172.21.15.107:0/1784430976 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5a74050a10 con 0x7f5a8c0ca2d0 2023-12-04T05:45:04.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.427+0000 7f5a9241e640 1 -- 172.21.15.107:0/1784430976 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a8c0c5810 con 0x7f5a8c0ca2d0 2023-12-04T05:45:04.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.427+0000 7f5a88ff9640 1 -- 172.21.15.107:0/1784430976 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5827 (secure 0 0 0) 0x7f5a74054030 con 0x7f5a8c0ca2d0 2023-12-04T05:45:04.431 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:04.431 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:44:42.900817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4747,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:stopping","state_seq":45,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4747":{"gid":4747,"name":"b","rank":1,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.195:6836/2222928136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2222928136},{"type":"v1","addr":"172.21.15.195:6836","nonce":2222928136}]},"join_fscid":-1,"laggy_since":"2023-12-04T05:43:59.753818+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:04.431 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 32 2023-12-04T05:45:04.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.431+0000 7f5a9241e640 1 -- 172.21.15.107:0/1784430976 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5a6c0668d0 msgr2=0x7f5a6c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:04.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.431+0000 7f5a9241e640 1 --2- 172.21.15.107:0/1784430976 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5a6c0668d0 0x7f5a6c068d90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c009af0 tx=0x7f5a7c008040 comp rx=0 tx=0).stop 2023-12-04T05:45:04.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.431+0000 7f5a9241e640 1 -- 172.21.15.107:0/1784430976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a8c0ca2d0 msgr2=0x7f5a8c1efa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:04.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.431+0000 7f5a9241e640 1 --2- 172.21.15.107:0/1784430976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a8c0ca2d0 0x7f5a8c1efa00 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5a7400c340 tx=0x7f5a7400c800 comp rx=0 tx=0).stop 2023-12-04T05:45:04.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.431+0000 7f5a9241e640 1 -- 172.21.15.107:0/1784430976 shutdown_connections 2023-12-04T05:45:04.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.431+0000 7f5a9241e640 1 --2- 172.21.15.107:0/1784430976 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5a6c0668d0 0x7f5a6c068d90 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:04.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.431+0000 7f5a9241e640 1 --2- 172.21.15.107:0/1784430976 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5a8c14f1c0 0x7f5a8c15b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:04.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.431+0000 7f5a9241e640 1 --2- 172.21.15.107:0/1784430976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a8c0ca2d0 0x7f5a8c1efa00 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:04.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.431+0000 7f5a9241e640 1 --2- 172.21.15.107:0/1784430976 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5a8c072ab0 0x7f5a8c146fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:04.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.431+0000 7f5a9241e640 1 -- 172.21.15.107:0/1784430976 >> 172.21.15.107:0/1784430976 conn(0x7f5a8c0663a0 msgr2=0x7f5a8c0cd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:04.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.431+0000 7f5a9241e640 1 -- 172.21.15.107:0/1784430976 shutdown_connections 2023-12-04T05:45:04.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:04.431+0000 7f5a9241e640 1 -- 172.21.15.107:0/1784430976 wait complete. 2023-12-04T05:45:04.450 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:45:06.451 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:06.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.579+0000 7fbc397fb640 1 Processor -- start 2023-12-04T05:45:06.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.583+0000 7fbc397fb640 1 -- start start 2023-12-04T05:45:06.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.583+0000 7fbc397fb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbc34072110 0x7fbc34158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:06.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.583+0000 7fbc397fb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fbc340c56f0 0x7fbc34140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:06.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.583+0000 7fbc397fb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc34158da0 con 0x7fbc340724f0 2023-12-04T05:45:06.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.583+0000 7fbc397fb640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc341583d0 con 0x7fbc340c56f0 2023-12-04T05:45:06.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.583+0000 7fbc397fb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc340c6170 con 0x7fbc34072110 2023-12-04T05:45:06.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.583+0000 7fbc337fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fbc340c56f0 0x7fbc34140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:06.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.583+0000 7fbc327fc640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fbc340724f0 0x7fbc340c7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:38094/0 (socket says 172.21.15.107:38094) 2023-12-04T05:45:06.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.583+0000 7fbc32ffd640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbc34072110 0x7fbc34158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:06.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.583+0000 7fbc327fc640 1 -- 172.21.15.107:0/2789527928 learned_addr learned my addr 172.21.15.107:0/2789527928 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:45:06.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.583+0000 7fbc31ffb640 1 -- 172.21.15.107:0/2789527928 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583749251 0 0) 0x7fbc1c001120 con 0x7fbc340724f0 2023-12-04T05:45:06.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.583+0000 7fbc31ffb640 1 -- 172.21.15.107:0/2789527928 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc20003530 con 0x7fbc340724f0 2023-12-04T05:45:06.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.583+0000 7fbc337fe640 1 -- 172.21.15.107:0/2789527928 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbc34072110 msgr2=0x7fbc34158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:06.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.583+0000 7fbc337fe640 1 --2- 172.21.15.107:0/2789527928 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbc34072110 0x7fbc34158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:06.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.583+0000 7fbc337fe640 1 -- 172.21.15.107:0/2789527928 >> v1:172.21.15.195:6789/0 conn(0x7fbc340724f0 legacy=0x7fbc340c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:06.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.583+0000 7fbc337fe640 1 -- 172.21.15.107:0/2789527928 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc34157da0 con 0x7fbc340c56f0 2023-12-04T05:45:06.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.583+0000 7fbc32ffd640 1 --2- 172.21.15.107:0/2789527928 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbc34072110 0x7fbc34158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T05:45:06.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.587+0000 7fbc337fe640 1 --2- 172.21.15.107:0/2789527928 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc340c56f0 0x7fbc34140830 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c0097c0 tx=0x7fbc2c01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b95a925eed17b7c7 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:06.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.587+0000 7fbc31ffb640 1 -- 172.21.15.107:0/2789527928 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c0ab600 con 0x7fbc340c56f0 2023-12-04T05:45:06.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.587+0000 7fbc31ffb640 1 -- 172.21.15.107:0/2789527928 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2c0a7050 con 0x7fbc340c56f0 2023-12-04T05:45:06.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.587+0000 7fbc31ffb640 1 -- 172.21.15.107:0/2789527928 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c0ab960 con 0x7fbc340c56f0 2023-12-04T05:45:06.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.587+0000 7fbc397fb640 1 -- 172.21.15.107:0/2789527928 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc340c56f0 msgr2=0x7fbc34140830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:06.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.587+0000 7fbc397fb640 1 --2- 172.21.15.107:0/2789527928 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc340c56f0 0x7fbc34140830 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c0097c0 tx=0x7fbc2c01c480 comp rx=0 tx=0).stop 2023-12-04T05:45:06.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.587+0000 7fbc397fb640 1 -- 172.21.15.107:0/2789527928 shutdown_connections 2023-12-04T05:45:06.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.587+0000 7fbc397fb640 1 --2- 172.21.15.107:0/2789527928 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc340c56f0 0x7fbc34140830 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:06.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.587+0000 7fbc397fb640 1 --2- 172.21.15.107:0/2789527928 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbc34072110 0x7fbc34158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:06.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.587+0000 7fbc397fb640 1 -- 172.21.15.107:0/2789527928 >> 172.21.15.107:0/2789527928 conn(0x7fbc340c7ff0 msgr2=0x7fbc340c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:06.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.587+0000 7fbc397fb640 1 -- 172.21.15.107:0/2789527928 shutdown_connections 2023-12-04T05:45:06.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.587+0000 7fbc397fb640 1 -- 172.21.15.107:0/2789527928 wait complete. 2023-12-04T05:45:06.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.587+0000 7fbc397fb640 1 Processor -- start 2023-12-04T05:45:06.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.587+0000 7fbc397fb640 1 -- start start 2023-12-04T05:45:06.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc397fb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc34072110 0x7fbc34162300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:06.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc397fb640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc340724f0 0x7fbc3415b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:06.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc397fb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbc340c56f0 0x7fbc3413fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:06.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc397fb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc34162840 con 0x7fbc34072110 2023-12-04T05:45:06.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc397fb640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc341f5960 con 0x7fbc340724f0 2023-12-04T05:45:06.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc397fb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc34152fd0 con 0x7fbc340c56f0 2023-12-04T05:45:06.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc32ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc34072110 0x7fbc34162300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:06.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc32ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc34072110 0x7fbc34162300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54708/0 (socket says 172.21.15.107:54708) 2023-12-04T05:45:06.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc32ffd640 1 -- 172.21.15.107:0/2762417266 learned_addr learned my addr 172.21.15.107:0/2762417266 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:06.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc327fc640 1 --2- 172.21.15.107:0/2762417266 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc340724f0 0x7fbc3415b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:06.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc337fe640 1 --2- 172.21.15.107:0/2762417266 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbc340c56f0 0x7fbc3413fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:06.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc32ffd640 1 -- 172.21.15.107:0/2762417266 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbc340c56f0 msgr2=0x7fbc3413fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:06.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc32ffd640 1 --2- 172.21.15.107:0/2762417266 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbc340c56f0 0x7fbc3413fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:06.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc32ffd640 1 -- 172.21.15.107:0/2762417266 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc340724f0 msgr2=0x7fbc3415b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:06.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc32ffd640 1 --2- 172.21.15.107:0/2762417266 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc340724f0 0x7fbc3415b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:06.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc32ffd640 1 -- 172.21.15.107:0/2762417266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc24009a70 con 0x7fbc34072110 2023-12-04T05:45:06.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc32ffd640 1 --2- 172.21.15.107:0/2762417266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc34072110 0x7fbc34162300 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fbc2400efc0 tx=0x7fbc2400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:06.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc2b7fe640 1 -- 172.21.15.107:0/2762417266 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc24014680 con 0x7fbc34072110 2023-12-04T05:45:06.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc2b7fe640 1 -- 172.21.15.107:0/2762417266 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2400f040 con 0x7fbc34072110 2023-12-04T05:45:06.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc397fb640 1 -- 172.21.15.107:0/2762417266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc2c009420 con 0x7fbc34072110 2023-12-04T05:45:06.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc2b7fe640 1 -- 172.21.15.107:0/2762417266 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc24014a00 con 0x7fbc34072110 2023-12-04T05:45:06.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc397fb640 1 -- 172.21.15.107:0/2762417266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc341f5bb0 con 0x7fbc34072110 2023-12-04T05:45:06.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc397fb640 1 -- 172.21.15.107:0/2762417266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc34067870 con 0x7fbc34072110 2023-12-04T05:45:06.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc2b7fe640 1 -- 172.21.15.107:0/2762417266 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fbc24014b60 con 0x7fbc34072110 2023-12-04T05:45:06.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc2b7fe640 1 --2- 172.21.15.107:0/2762417266 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbc140668d0 0x7fbc14068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:06.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc2b7fe640 1 -- 172.21.15.107:0/2762417266 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5997+0+0 (secure 0 0 0) 0x7fbc24085590 con 0x7fbc34072110 2023-12-04T05:45:06.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc327fc640 1 --2- 172.21.15.107:0/2762417266 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbc140668d0 0x7fbc14068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:06.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.591+0000 7fbc327fc640 1 --2- 172.21.15.107:0/2762417266 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbc140668d0 0x7fbc14068d90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbc3413f800 tx=0x7fbc1c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:06.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.595+0000 7fbc2b7fe640 1 -- 172.21.15.107:0/2762417266 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbc240526b0 con 0x7fbc34072110 2023-12-04T05:45:06.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.783+0000 7fbc397fb640 1 -- 172.21.15.107:0/2762417266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc34153650 con 0x7fbc34072110 2023-12-04T05:45:06.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.783+0000 7fbc2b7fe640 1 -- 172.21.15.107:0/2762417266 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5827 (secure 0 0 0) 0x7fbc24055cd0 con 0x7fbc34072110 2023-12-04T05:45:06.786 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:06.786 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:44:42.900817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4747,"mds_2":4263,"mds_3":4260,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"a","rank":4,"incarnation":18,"state":"up:stopping","state_seq":45,"addr":"172.21.15.107:6839/993693883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":993693883},{"type":"v1","addr":"172.21.15.107:6839","nonce":993693883}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4747":{"gid":4747,"name":"b","rank":1,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.195:6836/2222928136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":2222928136},{"type":"v1","addr":"172.21.15.195:6836","nonce":2222928136}]},"join_fscid":-1,"laggy_since":"2023-12-04T05:43:59.753818+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:06.786 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 32 2023-12-04T05:45:06.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.787+0000 7fbc397fb640 1 -- 172.21.15.107:0/2762417266 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbc140668d0 msgr2=0x7fbc14068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:06.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.787+0000 7fbc397fb640 1 --2- 172.21.15.107:0/2762417266 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbc140668d0 0x7fbc14068d90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbc3413f800 tx=0x7fbc1c009040 comp rx=0 tx=0).stop 2023-12-04T05:45:06.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.787+0000 7fbc397fb640 1 -- 172.21.15.107:0/2762417266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc34072110 msgr2=0x7fbc34162300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:06.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.787+0000 7fbc397fb640 1 --2- 172.21.15.107:0/2762417266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc34072110 0x7fbc34162300 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fbc2400efc0 tx=0x7fbc2400c490 comp rx=0 tx=0).stop 2023-12-04T05:45:06.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.787+0000 7fbc397fb640 1 -- 172.21.15.107:0/2762417266 shutdown_connections 2023-12-04T05:45:06.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.787+0000 7fbc397fb640 1 --2- 172.21.15.107:0/2762417266 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbc140668d0 0x7fbc14068d90 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:06.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.787+0000 7fbc397fb640 1 --2- 172.21.15.107:0/2762417266 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbc340c56f0 0x7fbc3413fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:06.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.787+0000 7fbc397fb640 1 --2- 172.21.15.107:0/2762417266 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc340724f0 0x7fbc3415b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:06.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.787+0000 7fbc397fb640 1 --2- 172.21.15.107:0/2762417266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc34072110 0x7fbc34162300 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:06.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.787+0000 7fbc397fb640 1 -- 172.21.15.107:0/2762417266 >> 172.21.15.107:0/2762417266 conn(0x7fbc340c7ff0 msgr2=0x7fbc340c92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:06.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.787+0000 7fbc397fb640 1 -- 172.21.15.107:0/2762417266 shutdown_connections 2023-12-04T05:45:06.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:06.787+0000 7fbc397fb640 1 -- 172.21.15.107:0/2762417266 wait complete. 2023-12-04T05:45:06.804 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-12-04T05:45:06.805 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 32, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-12-04T05:39:59.032019+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 32, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4247": { "addr": "172.21.15.107:6839/993693883", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6838", "nonce": 993693883, "type": "v2" }, { "addr": "172.21.15.107:6839", "nonce": 993693883, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 18, "join_fscid": -1, "name": "a", "rank": 4, "state": "up:stopping", "state_seq": 45 }, "gid_4260": { "addr": "172.21.15.107:6836/3444851515", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 3444851515, "type": "v2" }, { "addr": "172.21.15.107:6836", "nonce": 3444851515, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4260, "incarnation": 14, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 9 }, "gid_4263": { "addr": "172.21.15.107:6837/2210527429", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6835", "nonce": 2210527429, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 2210527429, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4263, "incarnation": 12, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 7 }, "gid_4288": { "addr": "172.21.15.195:6837/3412865477", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6835", "nonce": 3412865477, "type": "v2" }, { "addr": "172.21.15.195:6837", "nonce": 3412865477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4747": { "addr": "172.21.15.195:6836/2222928136", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6834", "nonce": 2222928136, "type": "v2" }, { "addr": "172.21.15.195:6836", "nonce": 2222928136, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4747, "incarnation": 25, "join_fscid": -1, "laggy_since": "2023-12-04T05:43:59.753818+0000", "name": "b", "rank": 1, "state": "up:active", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 25, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-12-04T05:44:42.900817+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 4747, "mds_2": 4263, "mds_3": 4260, "mds_4": 4247 } } } ], "standbys": [] } 2023-12-04T05:45:08.090 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-12-04T05:45:08.276 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~81s 2023-12-04T05:45:08.276 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~121s 2023-12-04T05:45:08.805 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:08.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152faac640 1 Processor -- start 2023-12-04T05:45:08.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152faac640 1 -- start start 2023-12-04T05:45:08.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152faac640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15280c5c60 0x7f15280c2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:08.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152faac640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15280c7a40 con 0x7f15280b1b80 2023-12-04T05:45:08.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152faac640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f152806c120 con 0x7f1528157e10 2023-12-04T05:45:08.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152faac640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15280c6b50 con 0x7f15280c5c60 2023-12-04T05:45:08.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152d848640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f15280b1b80 0x7f1528072680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41708/0 (socket says 172.21.15.107:41708) 2023-12-04T05:45:08.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152d848640 1 -- 172.21.15.107:0/2932425766 learned_addr learned my addr 172.21.15.107:0/2932425766 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:45:08.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152d047640 1 --2- 172.21.15.107:0/2932425766 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15280c5c60 0x7f15280c2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:08.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152c846640 1 -- 172.21.15.107:0/2932425766 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767900810 0 0) 0x7f15280c7a40 con 0x7f15280b1b80 2023-12-04T05:45:08.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152c846640 1 -- 172.21.15.107:0/2932425766 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1514003530 con 0x7f15280b1b80 2023-12-04T05:45:08.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152c846640 1 -- 172.21.15.107:0/2932425766 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 819238809 0 0) 0x7f1514003530 con 0x7f15280b1b80 2023-12-04T05:45:08.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152c846640 1 -- 172.21.15.107:0/2932425766 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15280c5c60 msgr2=0x7f15280c2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:08.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152c846640 1 --2- 172.21.15.107:0/2932425766 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15280c5c60 0x7f15280c2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:08.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152c846640 1 -- 172.21.15.107:0/2932425766 >> v1:172.21.15.195:6789/0 conn(0x7f1528157e10 legacy=0x7f15281626b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:08.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152c846640 1 -- 172.21.15.107:0/2932425766 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15280c7ce0 con 0x7f15280b1b80 2023-12-04T05:45:08.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152c846640 1 -- 172.21.15.107:0/2932425766 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f1510001a00 con 0x7f15280b1b80 2023-12-04T05:45:08.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152c846640 1 -- 172.21.15.107:0/2932425766 >> v1:172.21.15.107:6789/0 conn(0x7f15280b1b80 legacy=0x7f1528072680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:08.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152c846640 1 --2- 172.21.15.107:0/2932425766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15140039b0 0x7f1514003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:08.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152c846640 1 --2- 172.21.15.107:0/2932425766 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15140047d0 0x7f1514004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:08.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152c846640 1 --2- 172.21.15.107:0/2932425766 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15280c5c60 0x7f1514005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:08.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152c846640 1 -- 172.21.15.107:0/2932425766 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15280c6b50 con 0x7f15140039b0 2023-12-04T05:45:08.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152c846640 1 -- 172.21.15.107:0/2932425766 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151400bad0 con 0x7f15140047d0 2023-12-04T05:45:08.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152c846640 1 -- 172.21.15.107:0/2932425766 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151400bc90 con 0x7f15280c5c60 2023-12-04T05:45:08.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152d848640 1 --2- 172.21.15.107:0/2932425766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15140039b0 0x7f1514003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:08.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152d047640 1 --2- 172.21.15.107:0/2932425766 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15140047d0 0x7f1514004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:08.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152e049640 1 --2- 172.21.15.107:0/2932425766 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15280c5c60 0x7f1514005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:08.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152d848640 1 -- 172.21.15.107:0/2932425766 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15280c5c60 msgr2=0x7f1514005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:08.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152d848640 1 --2- 172.21.15.107:0/2932425766 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15280c5c60 0x7f1514005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:08.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152d848640 1 -- 172.21.15.107:0/2932425766 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15140047d0 msgr2=0x7f1514004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:08.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152d848640 1 --2- 172.21.15.107:0/2932425766 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15140047d0 0x7f1514004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:08.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152d848640 1 -- 172.21.15.107:0/2932425766 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1514003530 con 0x7f15140039b0 2023-12-04T05:45:08.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.935+0000 7f152e049640 1 --2- 172.21.15.107:0/2932425766 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15280c5c60 0x7f1514005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:45:08.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.939+0000 7f152d848640 1 --2- 172.21.15.107:0/2932425766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15140039b0 0x7f1514003dd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f151000a800 tx=0x7f151001fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=925310aacd8cb411 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:08.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.939+0000 7f152c846640 1 -- 172.21.15.107:0/2932425766 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15100aa670 con 0x7f15140039b0 2023-12-04T05:45:08.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.939+0000 7f152c846640 1 -- 172.21.15.107:0/2932425766 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15100aa7d0 con 0x7f15140039b0 2023-12-04T05:45:08.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.939+0000 7f152c846640 1 -- 172.21.15.107:0/2932425766 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15100aabc0 con 0x7f15140039b0 2023-12-04T05:45:08.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.939+0000 7f152faac640 1 -- 172.21.15.107:0/2932425766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15140039b0 msgr2=0x7f1514003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:08.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.939+0000 7f152faac640 1 --2- 172.21.15.107:0/2932425766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15140039b0 0x7f1514003dd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f151000a800 tx=0x7f151001fa40 comp rx=0 tx=0).stop 2023-12-04T05:45:08.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.939+0000 7f152e049640 1 -- 172.21.15.107:0/2932425766 reap_dead start 2023-12-04T05:45:08.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.939+0000 7f152faac640 1 -- 172.21.15.107:0/2932425766 shutdown_connections 2023-12-04T05:45:08.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.939+0000 7f152faac640 1 -- 172.21.15.107:0/2932425766 >> 172.21.15.107:0/2932425766 conn(0x7f152806e6f0 msgr2=0x7f15280c8580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:08.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.939+0000 7f152faac640 1 -- 172.21.15.107:0/2932425766 shutdown_connections 2023-12-04T05:45:08.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.939+0000 7f152faac640 1 -- 172.21.15.107:0/2932425766 wait complete. 2023-12-04T05:45:08.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.939+0000 7f152faac640 1 Processor -- start 2023-12-04T05:45:08.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.939+0000 7f152faac640 1 -- start start 2023-12-04T05:45:08.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f152faac640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15140039b0 0x7f15281ed730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:08.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f152faac640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15281576f0 0x7f152814dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:08.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f152faac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15280c27f0 0x7f15281e44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:08.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f152faac640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15281eccb0 con 0x7f15280c27f0 2023-12-04T05:45:08.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f152faac640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15281edc70 con 0x7f15140039b0 2023-12-04T05:45:08.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f152faac640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15280cc1d0 con 0x7f15281576f0 2023-12-04T05:45:08.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f152e049640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15280c27f0 0x7f15281e44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:08.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f152e049640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15280c27f0 0x7f15281e44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34376/0 (socket says 172.21.15.107:34376) 2023-12-04T05:45:08.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f152e049640 1 -- 172.21.15.107:0/875410064 learned_addr learned my addr 172.21.15.107:0/875410064 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:08.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f152d848640 1 --2- 172.21.15.107:0/875410064 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15140039b0 0x7f15281ed730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:08.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f152d047640 1 --2- 172.21.15.107:0/875410064 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15281576f0 0x7f152814dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:08.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f152e049640 1 -- 172.21.15.107:0/875410064 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15281576f0 msgr2=0x7f152814dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:08.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f152e049640 1 --2- 172.21.15.107:0/875410064 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15281576f0 0x7f152814dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:08.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f152e049640 1 -- 172.21.15.107:0/875410064 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15140039b0 msgr2=0x7f15281ed730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:08.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f152e049640 1 --2- 172.21.15.107:0/875410064 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15140039b0 0x7f15281ed730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:08.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f152e049640 1 -- 172.21.15.107:0/875410064 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15280cc4b0 con 0x7f15280c27f0 2023-12-04T05:45:08.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f152e049640 1 --2- 172.21.15.107:0/875410064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15280c27f0 0x7f15281e44e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f152806d7b0 tx=0x7f151800d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:08.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f15267fc640 1 -- 172.21.15.107:0/875410064 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151800b070 con 0x7f15280c27f0 2023-12-04T05:45:08.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f15267fc640 1 -- 172.21.15.107:0/875410064 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1518007030 con 0x7f15280c27f0 2023-12-04T05:45:08.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f152faac640 1 -- 172.21.15.107:0/875410064 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15280cc790 con 0x7f15280c27f0 2023-12-04T05:45:08.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f15267fc640 1 -- 172.21.15.107:0/875410064 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15180073e0 con 0x7f15280c27f0 2023-12-04T05:45:08.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f152faac640 1 -- 172.21.15.107:0/875410064 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15281ed210 con 0x7f15280c27f0 2023-12-04T05:45:08.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f152faac640 1 -- 172.21.15.107:0/875410064 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15280cc980 con 0x7f15280c27f0 2023-12-04T05:45:08.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f15267fc640 1 -- 172.21.15.107:0/875410064 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f1518007540 con 0x7f15280c27f0 2023-12-04T05:45:08.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f15267fc640 1 --2- 172.21.15.107:0/875410064 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1508066900 0x7f1508068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:08.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f152d848640 1 --2- 172.21.15.107:0/875410064 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1508066900 0x7f1508068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:08.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f15267fc640 1 -- 172.21.15.107:0/875410064 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f1518083f90 con 0x7f15280c27f0 2023-12-04T05:45:08.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.943+0000 7f152d848640 1 --2- 172.21.15.107:0/875410064 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1508066900 0x7f1508068dc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f15100b1000 tx=0x7f1510009050 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:08.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:08.947+0000 7f15267fc640 1 -- 172.21.15.107:0/875410064 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1518051140 con 0x7f15280c27f0 2023-12-04T05:45:09.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:09.131+0000 7f152faac640 1 -- 172.21.15.107:0/875410064 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15281ed430 con 0x7f15280c27f0 2023-12-04T05:45:09.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:09.131+0000 7f15267fc640 1 -- 172.21.15.107:0/875410064 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f1518054760 con 0x7f15280c27f0 2023-12-04T05:45:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:09.135 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:09.028556+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3],"up":{"mds_0":4288,"mds_1":5106,"mds_2":4263,"mds_3":4260},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:stopping","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6839/3319682203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":3319682203},{"type":"v1","addr":"172.21.15.107:6839","nonce":3319682203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:09.135 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 36 2023-12-04T05:45:09.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:09.135+0000 7f152faac640 1 -- 172.21.15.107:0/875410064 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1508066900 msgr2=0x7f1508068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:09.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:09.135+0000 7f152faac640 1 --2- 172.21.15.107:0/875410064 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1508066900 0x7f1508068dc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f15100b1000 tx=0x7f1510009050 comp rx=0 tx=0).stop 2023-12-04T05:45:09.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:09.135+0000 7f152faac640 1 -- 172.21.15.107:0/875410064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15280c27f0 msgr2=0x7f15281e44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:09.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:09.135+0000 7f152faac640 1 --2- 172.21.15.107:0/875410064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15280c27f0 0x7f15281e44e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f152806d7b0 tx=0x7f151800d890 comp rx=0 tx=0).stop 2023-12-04T05:45:09.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:09.135+0000 7f152faac640 1 -- 172.21.15.107:0/875410064 shutdown_connections 2023-12-04T05:45:09.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:09.135+0000 7f152faac640 1 --2- 172.21.15.107:0/875410064 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1508066900 0x7f1508068dc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:09.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:09.135+0000 7f152faac640 1 --2- 172.21.15.107:0/875410064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15280c27f0 0x7f15281e44e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:09.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:09.135+0000 7f152faac640 1 --2- 172.21.15.107:0/875410064 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f15281576f0 0x7f152814dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:09.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:09.135+0000 7f152faac640 1 --2- 172.21.15.107:0/875410064 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15140039b0 0x7f15281ed730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:09.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:09.135+0000 7f152faac640 1 -- 172.21.15.107:0/875410064 >> 172.21.15.107:0/875410064 conn(0x7f152806e6f0 msgr2=0x7f1528155a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:09.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:09.135+0000 7f152faac640 1 -- 172.21.15.107:0/875410064 shutdown_connections 2023-12-04T05:45:09.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:09.135+0000 7f152faac640 1 -- 172.21.15.107:0/875410064 wait complete. 2023-12-04T05:45:09.152 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:45:11.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:11.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.283+0000 7f9fe31bb640 1 Processor -- start 2023-12-04T05:45:11.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.283+0000 7f9fe31bb640 1 -- start start 2023-12-04T05:45:11.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.283+0000 7f9fe31bb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9fdc072110 0x7f9fdc158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:11.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.283+0000 7f9fe31bb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9fdc0c56f0 0x7f9fdc140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:11.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.283+0000 7f9fe31bb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fdc158da0 con 0x7f9fdc0724f0 2023-12-04T05:45:11.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.283+0000 7f9fe31bb640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fdc1583d0 con 0x7f9fdc0c56f0 2023-12-04T05:45:11.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.283+0000 7f9fe31bb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc0c6170 con 0x7f9fdc072110 2023-12-04T05:45:11.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.283+0000 7f9fe1758640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9fdc0c56f0 0x7f9fdc140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:11.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.283+0000 7f9fe0f57640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9fdc072110 0x7f9fdc158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:11.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.283+0000 7f9fe1758640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9fdc0c56f0 0x7f9fdc140830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.107:60694/0 (socket says 172.21.15.107:60694) 2023-12-04T05:45:11.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.283+0000 7f9fe1758640 1 -- 172.21.15.107:0/2711588510 learned_addr learned my addr 172.21.15.107:0/2711588510 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:11.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.283+0000 7f9fdbfff640 1 --1- 172.21.15.107:0/2711588510 >> v1:172.21.15.195:6789/0 conn(0x7f9fdc0724f0 0x7f9fdc0c7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:60530/0 (socket says 172.21.15.107:60530) 2023-12-04T05:45:11.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.287+0000 7f9fe0f57640 1 -- 172.21.15.107:0/2711588510 >> v2:172.21.15.195:3300/0 conn(0x7f9fdc0c56f0 msgr2=0x7f9fdc140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:11.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.287+0000 7f9fe0f57640 1 --2- 172.21.15.107:0/2711588510 >> v2:172.21.15.195:3300/0 conn(0x7f9fdc0c56f0 0x7f9fdc140830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:11.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.287+0000 7f9fe0f57640 1 -- 172.21.15.107:0/2711588510 >> v1:172.21.15.195:6789/0 conn(0x7f9fdc0724f0 legacy=0x7f9fdc0c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:11.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.287+0000 7f9fe0f57640 1 -- 172.21.15.107:0/2711588510 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fdc157da0 con 0x7f9fdc072110 2023-12-04T05:45:11.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.287+0000 7f9fe1758640 1 --2- 172.21.15.107:0/2711588510 >> v2:172.21.15.195:3300/0 conn(0x7f9fdc0c56f0 0x7f9fdc140830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T05:45:11.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.287+0000 7f9fe0f57640 1 --2- 172.21.15.107:0/2711588510 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9fdc072110 0x7f9fdc158860 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc0097c0 tx=0x7f9fcc01c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8a8f77fd781dc33 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:11.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.287+0000 7f9fdb7fe640 1 -- 172.21.15.107:0/2711588510 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc0ab600 con 0x7f9fdc072110 2023-12-04T05:45:11.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.287+0000 7f9fdb7fe640 1 -- 172.21.15.107:0/2711588510 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fcc0a7050 con 0x7f9fdc072110 2023-12-04T05:45:11.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.287+0000 7f9fdb7fe640 1 -- 172.21.15.107:0/2711588510 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc0ab980 con 0x7f9fdc072110 2023-12-04T05:45:11.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.287+0000 7f9fe31bb640 1 -- 172.21.15.107:0/2711588510 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9fdc072110 msgr2=0x7f9fdc158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:11.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.287+0000 7f9fe31bb640 1 --2- 172.21.15.107:0/2711588510 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9fdc072110 0x7f9fdc158860 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc0097c0 tx=0x7f9fcc01c480 comp rx=0 tx=0).stop 2023-12-04T05:45:11.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.287+0000 7f9fe31bb640 1 -- 172.21.15.107:0/2711588510 shutdown_connections 2023-12-04T05:45:11.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.287+0000 7f9fe31bb640 1 --2- 172.21.15.107:0/2711588510 >> v2:172.21.15.195:3300/0 conn(0x7f9fdc0c56f0 0x7f9fdc140830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:11.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.287+0000 7f9fe31bb640 1 --2- 172.21.15.107:0/2711588510 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9fdc072110 0x7f9fdc158860 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:11.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.287+0000 7f9fe31bb640 1 -- 172.21.15.107:0/2711588510 >> 172.21.15.107:0/2711588510 conn(0x7f9fdc0c7ff0 msgr2=0x7f9fdc0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:11.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.287+0000 7f9fe31bb640 1 -- 172.21.15.107:0/2711588510 shutdown_connections 2023-12-04T05:45:11.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.287+0000 7f9fe31bb640 1 -- 172.21.15.107:0/2711588510 wait complete. 2023-12-04T05:45:11.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fe31bb640 1 Processor -- start 2023-12-04T05:45:11.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fe31bb640 1 -- start start 2023-12-04T05:45:11.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fe31bb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9fdc072110 0x7f9fdc1364e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:11.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fe31bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fdc0724f0 0x7f9fdc137a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:11.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fe31bb640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9fdc0c56f0 0x7f9fdc1f2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:11.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fe31bb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc1345e0 con 0x7f9fdc0724f0 2023-12-04T05:45:11.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fe31bb640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc135bb0 con 0x7f9fdc0c56f0 2023-12-04T05:45:11.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fe31bb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc135de0 con 0x7f9fdc072110 2023-12-04T05:45:11.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fdbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fdc0724f0 0x7f9fdc137a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:11.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fdbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fdc0724f0 0x7f9fdc137a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34380/0 (socket says 172.21.15.107:34380) 2023-12-04T05:45:11.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fdbfff640 1 -- 172.21.15.107:0/1485277461 learned_addr learned my addr 172.21.15.107:0/1485277461 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:11.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fe0f57640 1 --2- 172.21.15.107:0/1485277461 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9fdc072110 0x7f9fdc1364e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:11.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fe1758640 1 --2- 172.21.15.107:0/1485277461 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9fdc0c56f0 0x7f9fdc1f2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:11.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fdbfff640 1 -- 172.21.15.107:0/1485277461 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9fdc072110 msgr2=0x7f9fdc1364e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:11.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fdbfff640 1 --2- 172.21.15.107:0/1485277461 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9fdc072110 0x7f9fdc1364e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:11.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fdbfff640 1 -- 172.21.15.107:0/1485277461 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9fdc0c56f0 msgr2=0x7f9fdc1f2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:11.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fdbfff640 1 --2- 172.21.15.107:0/1485277461 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9fdc0c56f0 0x7f9fdc1f2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:11.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fdbfff640 1 -- 172.21.15.107:0/1485277461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd4009b30 con 0x7f9fdc0724f0 2023-12-04T05:45:11.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fe0f57640 1 --2- 172.21.15.107:0/1485277461 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9fdc072110 0x7f9fdc1364e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:45:11.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fdbfff640 1 --2- 172.21.15.107:0/1485277461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fdc0724f0 0x7f9fdc137a30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9fd000c460 tx=0x7f9fd000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:11.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fd97fa640 1 -- 172.21.15.107:0/1485277461 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd0007480 con 0x7f9fdc0724f0 2023-12-04T05:45:11.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fd97fa640 1 -- 172.21.15.107:0/1485277461 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd00075e0 con 0x7f9fdc0724f0 2023-12-04T05:45:11.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fe31bb640 1 -- 172.21.15.107:0/1485277461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fcc009420 con 0x7f9fdc0724f0 2023-12-04T05:45:11.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fd97fa640 1 -- 172.21.15.107:0/1485277461 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd0004a30 con 0x7f9fdc0724f0 2023-12-04T05:45:11.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.291+0000 7f9fe31bb640 1 -- 172.21.15.107:0/1485277461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fdc1f1ca0 con 0x7f9fdc0724f0 2023-12-04T05:45:11.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.295+0000 7f9fe31bb640 1 -- 172.21.15.107:0/1485277461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fdc067870 con 0x7f9fdc0724f0 2023-12-04T05:45:11.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.295+0000 7f9fd97fa640 1 -- 172.21.15.107:0/1485277461 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9fd0004b90 con 0x7f9fdc0724f0 2023-12-04T05:45:11.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.295+0000 7f9fd97fa640 1 --2- 172.21.15.107:0/1485277461 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9fc8066900 0x7f9fc8068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:11.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.295+0000 7f9fe0f57640 1 --2- 172.21.15.107:0/1485277461 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9fc8066900 0x7f9fc8068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:11.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.295+0000 7f9fd97fa640 1 -- 172.21.15.107:0/1485277461 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f9fd0083880 con 0x7f9fdc0724f0 2023-12-04T05:45:11.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.295+0000 7f9fe0f57640 1 --2- 172.21.15.107:0/1485277461 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9fc8066900 0x7f9fc8068dc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc0ad000 tx=0x7f9fcc0a6000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:11.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.299+0000 7f9fd97fa640 1 -- 172.21.15.107:0/1485277461 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9fd00508b0 con 0x7f9fdc0724f0 2023-12-04T05:45:11.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.483+0000 7f9fe31bb640 1 -- 172.21.15.107:0/1485277461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fdc158da0 con 0x7f9fdc0724f0 2023-12-04T05:45:11.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.483+0000 7f9fd97fa640 1 -- 172.21.15.107:0/1485277461 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f9fdc158da0 con 0x7f9fdc0724f0 2023-12-04T05:45:11.486 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:11.487 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:09.028556+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3],"up":{"mds_0":4288,"mds_1":5106,"mds_2":4263,"mds_3":4260},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:stopping","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6839/3319682203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":3319682203},{"type":"v1","addr":"172.21.15.107:6839","nonce":3319682203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:11.487 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 36 2023-12-04T05:45:11.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.487+0000 7f9fe31bb640 1 -- 172.21.15.107:0/1485277461 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9fc8066900 msgr2=0x7f9fc8068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:11.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.487+0000 7f9fe31bb640 1 --2- 172.21.15.107:0/1485277461 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9fc8066900 0x7f9fc8068dc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc0ad000 tx=0x7f9fcc0a6000 comp rx=0 tx=0).stop 2023-12-04T05:45:11.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.487+0000 7f9fe31bb640 1 -- 172.21.15.107:0/1485277461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fdc0724f0 msgr2=0x7f9fdc137a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:11.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.487+0000 7f9fe31bb640 1 --2- 172.21.15.107:0/1485277461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fdc0724f0 0x7f9fdc137a30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9fd000c460 tx=0x7f9fd000c920 comp rx=0 tx=0).stop 2023-12-04T05:45:11.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.487+0000 7f9fe31bb640 1 -- 172.21.15.107:0/1485277461 shutdown_connections 2023-12-04T05:45:11.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.487+0000 7f9fe31bb640 1 --2- 172.21.15.107:0/1485277461 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9fc8066900 0x7f9fc8068dc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:11.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.487+0000 7f9fe31bb640 1 --2- 172.21.15.107:0/1485277461 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9fdc0c56f0 0x7f9fdc1f2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:11.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.487+0000 7f9fe31bb640 1 --2- 172.21.15.107:0/1485277461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fdc0724f0 0x7f9fdc137a30 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:11.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.487+0000 7f9fe31bb640 1 --2- 172.21.15.107:0/1485277461 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9fdc072110 0x7f9fdc1364e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:11.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.487+0000 7f9fe31bb640 1 -- 172.21.15.107:0/1485277461 >> 172.21.15.107:0/1485277461 conn(0x7f9fdc0c7ff0 msgr2=0x7f9fdc155bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:11.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.487+0000 7f9fe31bb640 1 -- 172.21.15.107:0/1485277461 shutdown_connections 2023-12-04T05:45:11.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:11.487+0000 7f9fe31bb640 1 -- 172.21.15.107:0/1485277461 wait complete. 2023-12-04T05:45:11.504 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:45:13.505 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:13.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.635+0000 7f9d44066640 1 Processor -- start 2023-12-04T05:45:13.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.635+0000 7f9d44066640 1 -- start start 2023-12-04T05:45:13.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.635+0000 7f9d44066640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d3c0c5ae0 0x7f9d3c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:13.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.635+0000 7f9d44066640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9d3c157730 0x7f9d3c06bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:13.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.635+0000 7f9d44066640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d3c0c6380 con 0x7f9d3c0c52c0 2023-12-04T05:45:13.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.635+0000 7f9d44066640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d3c071100 con 0x7f9d3c157730 2023-12-04T05:45:13.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.635+0000 7f9d44066640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c0c56a0 con 0x7f9d3c0c5ae0 2023-12-04T05:45:13.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.635+0000 7f9d42603640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9d3c157730 0x7f9d3c06bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:13.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.635+0000 7f9d41601640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d3c0c5ae0 0x7f9d3c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:13.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.635+0000 7f9d41e02640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f9d3c0c52c0 0x7f9d3c0721a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:60546/0 (socket says 172.21.15.107:60546) 2023-12-04T05:45:13.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.635+0000 7f9d41e02640 1 -- 172.21.15.107:0/757080553 learned_addr learned my addr 172.21.15.107:0/757080553 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:45:13.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.635+0000 7f9d42603640 1 -- 172.21.15.107:0/757080553 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d3c0c5ae0 msgr2=0x7f9d3c06b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:13.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.635+0000 7f9d42603640 1 --2- 172.21.15.107:0/757080553 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d3c0c5ae0 0x7f9d3c06b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:13.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.635+0000 7f9d42603640 1 -- 172.21.15.107:0/757080553 >> v1:172.21.15.195:6789/0 conn(0x7f9d3c0c52c0 legacy=0x7f9d3c0721a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:13.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.635+0000 7f9d42603640 1 -- 172.21.15.107:0/757080553 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3c070b40 con 0x7f9d3c157730 2023-12-04T05:45:13.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.639+0000 7f9d42603640 1 --2- 172.21.15.107:0/757080553 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d3c157730 0x7f9d3c06bc00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9d34009d90 tx=0x7f9d3401c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fbfa23abd697846 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:13.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.639+0000 7f9d40e00640 1 -- 172.21.15.107:0/757080553 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d340a5900 con 0x7f9d3c157730 2023-12-04T05:45:13.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.639+0000 7f9d40e00640 1 -- 172.21.15.107:0/757080553 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d340a5a60 con 0x7f9d3c157730 2023-12-04T05:45:13.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.639+0000 7f9d40e00640 1 -- 172.21.15.107:0/757080553 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d340a5df0 con 0x7f9d3c157730 2023-12-04T05:45:13.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.639+0000 7f9d44066640 1 -- 172.21.15.107:0/757080553 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d3c157730 msgr2=0x7f9d3c06bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:13.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.639+0000 7f9d44066640 1 --2- 172.21.15.107:0/757080553 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d3c157730 0x7f9d3c06bc00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9d34009d90 tx=0x7f9d3401c950 comp rx=0 tx=0).stop 2023-12-04T05:45:13.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.639+0000 7f9d44066640 1 -- 172.21.15.107:0/757080553 shutdown_connections 2023-12-04T05:45:13.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.639+0000 7f9d44066640 1 --2- 172.21.15.107:0/757080553 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d3c157730 0x7f9d3c06bc00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:13.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.639+0000 7f9d44066640 1 --2- 172.21.15.107:0/757080553 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d3c0c5ae0 0x7f9d3c06b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:13.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.639+0000 7f9d44066640 1 -- 172.21.15.107:0/757080553 >> 172.21.15.107:0/757080553 conn(0x7f9d3c06e6f0 msgr2=0x7f9d3c0c8580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:13.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.639+0000 7f9d44066640 1 -- 172.21.15.107:0/757080553 shutdown_connections 2023-12-04T05:45:13.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.639+0000 7f9d44066640 1 -- 172.21.15.107:0/757080553 wait complete. 2023-12-04T05:45:13.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.639+0000 7f9d44066640 1 Processor -- start 2023-12-04T05:45:13.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.639+0000 7f9d44066640 1 -- start start 2023-12-04T05:45:13.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d44066640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d3c0c52c0 0x7f9d3c1497f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:13.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d44066640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d3c0c5ae0 0x7f9d3c06a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:13.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d44066640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d3c157730 0x7f9d3c14b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:13.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d44066640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c1544d0 con 0x7f9d3c0c52c0 2023-12-04T05:45:13.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d44066640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c14b510 con 0x7f9d3c157730 2023-12-04T05:45:13.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d44066640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c0c2530 con 0x7f9d3c0c5ae0 2023-12-04T05:45:13.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d41e02640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d3c0c52c0 0x7f9d3c1497f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:13.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d41e02640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d3c0c52c0 0x7f9d3c1497f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34384/0 (socket says 172.21.15.107:34384) 2023-12-04T05:45:13.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d41e02640 1 -- 172.21.15.107:0/3481298000 learned_addr learned my addr 172.21.15.107:0/3481298000 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:13.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d41601640 1 --2- 172.21.15.107:0/3481298000 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d3c0c5ae0 0x7f9d3c06a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:13.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d42603640 1 --2- 172.21.15.107:0/3481298000 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d3c157730 0x7f9d3c14b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:13.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d41e02640 1 -- 172.21.15.107:0/3481298000 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d3c0c5ae0 msgr2=0x7f9d3c06a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:13.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d41e02640 1 --2- 172.21.15.107:0/3481298000 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d3c0c5ae0 0x7f9d3c06a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:13.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d41e02640 1 -- 172.21.15.107:0/3481298000 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d3c157730 msgr2=0x7f9d3c14b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:13.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d41e02640 1 --2- 172.21.15.107:0/3481298000 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d3c157730 0x7f9d3c14b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:13.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d41e02640 1 -- 172.21.15.107:0/3481298000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d34009a70 con 0x7f9d3c0c52c0 2023-12-04T05:45:13.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d41e02640 1 --2- 172.21.15.107:0/3481298000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d3c0c52c0 0x7f9d3c1497f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f9d3800c420 tx=0x7f9d3800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:13.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d327fc640 1 -- 172.21.15.107:0/3481298000 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d38005cf0 con 0x7f9d3c0c52c0 2023-12-04T05:45:13.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d327fc640 1 -- 172.21.15.107:0/3481298000 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d38005e50 con 0x7f9d3c0c52c0 2023-12-04T05:45:13.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d44066640 1 -- 172.21.15.107:0/3481298000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d3c149d30 con 0x7f9d3c0c52c0 2023-12-04T05:45:13.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d327fc640 1 -- 172.21.15.107:0/3481298000 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d380072f0 con 0x7f9d3c0c52c0 2023-12-04T05:45:13.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d44066640 1 -- 172.21.15.107:0/3481298000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d3c1f3ad0 con 0x7f9d3c0c52c0 2023-12-04T05:45:13.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d44066640 1 -- 172.21.15.107:0/3481298000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d3c067ec0 con 0x7f9d3c0c52c0 2023-12-04T05:45:13.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d327fc640 1 -- 172.21.15.107:0/3481298000 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9d38007450 con 0x7f9d3c0c52c0 2023-12-04T05:45:13.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d327fc640 1 --2- 172.21.15.107:0/3481298000 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9d200668d0 0x7f9d20068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:13.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d327fc640 1 -- 172.21.15.107:0/3481298000 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f9d38083b30 con 0x7f9d3c0c52c0 2023-12-04T05:45:13.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d41601640 1 --2- 172.21.15.107:0/3481298000 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9d200668d0 0x7f9d20068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:13.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.643+0000 7f9d41601640 1 --2- 172.21.15.107:0/3481298000 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9d200668d0 0x7f9d20068d90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c1eae40 tx=0x7f9d2c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:13.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.647+0000 7f9d327fc640 1 -- 172.21.15.107:0/3481298000 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9d38050c80 con 0x7f9d3c0c52c0 2023-12-04T05:45:13.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.831+0000 7f9d44066640 1 -- 172.21.15.107:0/3481298000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d3c0b1f70 con 0x7f9d3c0c52c0 2023-12-04T05:45:13.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.835+0000 7f9d327fc640 1 -- 172.21.15.107:0/3481298000 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f9d38017070 con 0x7f9d3c0c52c0 2023-12-04T05:45:13.836 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:13.836 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:09.028556+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3],"up":{"mds_0":4288,"mds_1":5106,"mds_2":4263,"mds_3":4260},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:stopping","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6839/3319682203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":3319682203},{"type":"v1","addr":"172.21.15.107:6839","nonce":3319682203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:13.836 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 36 2023-12-04T05:45:13.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.835+0000 7f9d44066640 1 -- 172.21.15.107:0/3481298000 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9d200668d0 msgr2=0x7f9d20068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:13.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.835+0000 7f9d44066640 1 --2- 172.21.15.107:0/3481298000 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9d200668d0 0x7f9d20068d90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c1eae40 tx=0x7f9d2c008040 comp rx=0 tx=0).stop 2023-12-04T05:45:13.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.835+0000 7f9d44066640 1 -- 172.21.15.107:0/3481298000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d3c0c52c0 msgr2=0x7f9d3c1497f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:13.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.835+0000 7f9d44066640 1 --2- 172.21.15.107:0/3481298000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d3c0c52c0 0x7f9d3c1497f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f9d3800c420 tx=0x7f9d3800c8e0 comp rx=0 tx=0).stop 2023-12-04T05:45:13.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.835+0000 7f9d44066640 1 -- 172.21.15.107:0/3481298000 shutdown_connections 2023-12-04T05:45:13.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.835+0000 7f9d44066640 1 --2- 172.21.15.107:0/3481298000 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9d200668d0 0x7f9d20068d90 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:13.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.835+0000 7f9d44066640 1 --2- 172.21.15.107:0/3481298000 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d3c157730 0x7f9d3c14b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:13.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.835+0000 7f9d44066640 1 --2- 172.21.15.107:0/3481298000 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d3c0c5ae0 0x7f9d3c06a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:13.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.835+0000 7f9d44066640 1 --2- 172.21.15.107:0/3481298000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d3c0c52c0 0x7f9d3c1497f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:13.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.835+0000 7f9d44066640 1 -- 172.21.15.107:0/3481298000 >> 172.21.15.107:0/3481298000 conn(0x7f9d3c06e6f0 msgr2=0x7f9d3c06b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:13.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.835+0000 7f9d44066640 1 -- 172.21.15.107:0/3481298000 shutdown_connections 2023-12-04T05:45:13.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:13.835+0000 7f9d44066640 1 -- 172.21.15.107:0/3481298000 wait complete. 2023-12-04T05:45:13.855 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:45:14.993 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~87s 2023-12-04T05:45:14.993 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~128s 2023-12-04T05:45:15.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:15.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81ec4eb640 1 Processor -- start 2023-12-04T05:45:15.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81ec4eb640 1 -- start start 2023-12-04T05:45:15.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81ec4eb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81e4072110 0x7f81e4158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:15.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81ec4eb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e4158da0 con 0x7f81e40c56f0 2023-12-04T05:45:15.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81ec4eb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e41583d0 con 0x7f81e40724f0 2023-12-04T05:45:15.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81ec4eb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e40c6170 con 0x7f81e4072110 2023-12-04T05:45:15.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81eaa88640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f81e40c56f0 0x7f81e4140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41712/0 (socket says 172.21.15.107:41712) 2023-12-04T05:45:15.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81eaa88640 1 -- 172.21.15.107:0/3538660495 learned_addr learned my addr 172.21.15.107:0/3538660495 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:45:15.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81ea287640 1 --2- 172.21.15.107:0/3538660495 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81e4072110 0x7f81e4158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:15.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81e9285640 1 -- 172.21.15.107:0/3538660495 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 208593773 0 0) 0x7f81dc0014a0 con 0x7f81e40c56f0 2023-12-04T05:45:15.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81e9285640 1 -- 172.21.15.107:0/3538660495 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81d0003530 con 0x7f81e40c56f0 2023-12-04T05:45:15.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81e9285640 1 -- 172.21.15.107:0/3538660495 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2438784756 0 0) 0x7f81d0003530 con 0x7f81e40c56f0 2023-12-04T05:45:15.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81e9285640 1 -- 172.21.15.107:0/3538660495 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81e4072110 msgr2=0x7f81e4158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:15.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81e9285640 1 --2- 172.21.15.107:0/3538660495 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81e4072110 0x7f81e4158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:15.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81e9285640 1 -- 172.21.15.107:0/3538660495 >> v1:172.21.15.195:6789/0 conn(0x7f81e40724f0 legacy=0x7f81e40c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:15.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81e9285640 1 -- 172.21.15.107:0/3538660495 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e4157da0 con 0x7f81e40c56f0 2023-12-04T05:45:15.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81e9285640 1 -- 172.21.15.107:0/3538660495 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f81dc001b90 con 0x7f81e40c56f0 2023-12-04T05:45:15.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81e9285640 1 -- 172.21.15.107:0/3538660495 >> v1:172.21.15.107:6789/0 conn(0x7f81e40c56f0 legacy=0x7f81e4140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:15.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81e9285640 1 --2- 172.21.15.107:0/3538660495 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81e4072110 0x7f81d00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:15.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81e9285640 1 --2- 172.21.15.107:0/3538660495 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81d0003f30 0x7f81d00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81e9285640 1 --2- 172.21.15.107:0/3538660495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81d0004d10 0x7f81d0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81e9285640 1 -- 172.21.15.107:0/3538660495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e40c6170 con 0x7f81d0004d10 2023-12-04T05:45:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81e9285640 1 -- 172.21.15.107:0/3538660495 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d000ba70 con 0x7f81d0003f30 2023-12-04T05:45:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81e9285640 1 -- 172.21.15.107:0/3538660495 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d000bc30 con 0x7f81e4072110 2023-12-04T05:45:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81eaa88640 1 --2- 172.21.15.107:0/3538660495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81d0004d10 0x7f81d0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81e9a86640 1 --2- 172.21.15.107:0/3538660495 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81d0003f30 0x7f81d00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81ea287640 1 --2- 172.21.15.107:0/3538660495 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81e4072110 0x7f81d00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81eaa88640 1 -- 172.21.15.107:0/3538660495 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81e4072110 msgr2=0x7f81d00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81eaa88640 1 --2- 172.21.15.107:0/3538660495 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81e4072110 0x7f81d00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81eaa88640 1 -- 172.21.15.107:0/3538660495 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81d0003f30 msgr2=0x7f81d00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81eaa88640 1 --2- 172.21.15.107:0/3538660495 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81d0003f30 0x7f81d00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81eaa88640 1 -- 172.21.15.107:0/3538660495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d0003530 con 0x7f81d0004d10 2023-12-04T05:45:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.987+0000 7f81ea287640 1 --2- 172.21.15.107:0/3538660495 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81e4072110 0x7f81d00039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:45:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81eaa88640 1 --2- 172.21.15.107:0/3538660495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81d0004d10 0x7f81d0005190 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f81dc00a830 tx=0x7f81dc01d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b404123e1e9f6aa server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81e9285640 1 -- 172.21.15.107:0/3538660495 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc0ad630 con 0x7f81d0004d10 2023-12-04T05:45:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81e9285640 1 -- 172.21.15.107:0/3538660495 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81dc0a8070 con 0x7f81d0004d10 2023-12-04T05:45:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81e9285640 1 -- 172.21.15.107:0/3538660495 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc0ada30 con 0x7f81d0004d10 2023-12-04T05:45:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81ec4eb640 1 -- 172.21.15.107:0/3538660495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81d0004d10 msgr2=0x7f81d0005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81ec4eb640 1 --2- 172.21.15.107:0/3538660495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81d0004d10 0x7f81d0005190 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f81dc00a830 tx=0x7f81dc01d340 comp rx=0 tx=0).stop 2023-12-04T05:45:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81eaa88640 1 -- 172.21.15.107:0/3538660495 reap_dead start 2023-12-04T05:45:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81ec4eb640 1 -- 172.21.15.107:0/3538660495 shutdown_connections 2023-12-04T05:45:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81ec4eb640 1 -- 172.21.15.107:0/3538660495 >> 172.21.15.107:0/3538660495 conn(0x7f81e40c7ff0 msgr2=0x7f81e40c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81ec4eb640 1 -- 172.21.15.107:0/3538660495 shutdown_connections 2023-12-04T05:45:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81ec4eb640 1 -- 172.21.15.107:0/3538660495 wait complete. 2023-12-04T05:45:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81ec4eb640 1 Processor -- start 2023-12-04T05:45:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81ec4eb640 1 -- start start 2023-12-04T05:45:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81ec4eb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81d0004d10 0x7f81e41528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81ec4eb640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81e40b9970 0x7f81e40b9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81ec4eb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81e415b400 0x7f81e415b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81ec4eb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e41edb40 con 0x7f81e415b400 2023-12-04T05:45:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81ec4eb640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e4153e00 con 0x7f81e40b9970 2023-12-04T05:45:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81ec4eb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e415af70 con 0x7f81d0004d10 2023-12-04T05:45:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81eaa88640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81e415b400 0x7f81e415b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81eaa88640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81e415b400 0x7f81e415b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34400/0 (socket says 172.21.15.107:34400) 2023-12-04T05:45:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81eaa88640 1 -- 172.21.15.107:0/277521321 learned_addr learned my addr 172.21.15.107:0/277521321 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.991+0000 7f81ea287640 1 --2- 172.21.15.107:0/277521321 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81d0004d10 0x7f81e41528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.995+0000 7f81e9a86640 1 --2- 172.21.15.107:0/277521321 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81e40b9970 0x7f81e40b9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.995+0000 7f81eaa88640 1 -- 172.21.15.107:0/277521321 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81d0004d10 msgr2=0x7f81e41528f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.995+0000 7f81eaa88640 1 --2- 172.21.15.107:0/277521321 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81d0004d10 0x7f81e41528f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:15.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.995+0000 7f81eaa88640 1 -- 172.21.15.107:0/277521321 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81e40b9970 msgr2=0x7f81e40b9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:15.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.995+0000 7f81eaa88640 1 --2- 172.21.15.107:0/277521321 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81e40b9970 0x7f81e40b9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:15.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.995+0000 7f81eaa88640 1 -- 172.21.15.107:0/277521321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e415a360 con 0x7f81e415b400 2023-12-04T05:45:15.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.995+0000 7f81eaa88640 1 --2- 172.21.15.107:0/277521321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81e415b400 0x7f81e415b880 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f81dc01dbe0 tx=0x7f81d0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:15.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.995+0000 7f81daffd640 1 -- 172.21.15.107:0/277521321 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc0014a0 con 0x7f81e415b400 2023-12-04T05:45:15.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.995+0000 7f81daffd640 1 -- 172.21.15.107:0/277521321 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81dc004b70 con 0x7f81e415b400 2023-12-04T05:45:15.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.995+0000 7f81ec4eb640 1 -- 172.21.15.107:0/277521321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e415a640 con 0x7f81e415b400 2023-12-04T05:45:15.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.995+0000 7f81daffd640 1 -- 172.21.15.107:0/277521321 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc0ad430 con 0x7f81e415b400 2023-12-04T05:45:15.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.995+0000 7f81ec4eb640 1 -- 172.21.15.107:0/277521321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e415ac00 con 0x7f81e415b400 2023-12-04T05:45:15.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.995+0000 7f81ec4eb640 1 -- 172.21.15.107:0/277521321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e415ad90 con 0x7f81e415b400 2023-12-04T05:45:15.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.995+0000 7f81daffd640 1 -- 172.21.15.107:0/277521321 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f81dc0aa030 con 0x7f81e415b400 2023-12-04T05:45:15.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.995+0000 7f81daffd640 1 --2- 172.21.15.107:0/277521321 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f81c80668d0 0x7f81c8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:15.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.995+0000 7f81daffd640 1 -- 172.21.15.107:0/277521321 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f81dc11e710 con 0x7f81e415b400 2023-12-04T05:45:15.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.995+0000 7f81ea287640 1 --2- 172.21.15.107:0/277521321 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f81c80668d0 0x7f81c8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:15.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.995+0000 7f81ea287640 1 --2- 172.21.15.107:0/277521321 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f81c80668d0 0x7f81c8068d90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f81e0009ac0 tx=0x7f81e0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:16.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:15.999+0000 7f81daffd640 1 -- 172.21.15.107:0/277521321 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f81dc0eb860 con 0x7f81e415b400 2023-12-04T05:45:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:16.187+0000 7f81ec4eb640 1 -- 172.21.15.107:0/277521321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81e4158da0 con 0x7f81e415b400 2023-12-04T05:45:16.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:16.187+0000 7f81daffd640 1 -- 172.21.15.107:0/277521321 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f81e4158da0 con 0x7f81e415b400 2023-12-04T05:45:16.191 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:16.191 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:14.058884+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3],"up":{"mds_0":4288,"mds_1":5106,"mds_2":4263,"mds_3":4260},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:stopping","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":36,"state":"up:resolve","state_seq":3,"addr":"172.21.15.107:6839/3319682203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":3319682203},{"type":"v1","addr":"172.21.15.107:6839","nonce":3319682203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:16.192 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 37 2023-12-04T05:45:16.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:16.191+0000 7f81ec4eb640 1 -- 172.21.15.107:0/277521321 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f81c80668d0 msgr2=0x7f81c8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:16.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:16.191+0000 7f81ec4eb640 1 --2- 172.21.15.107:0/277521321 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f81c80668d0 0x7f81c8068d90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f81e0009ac0 tx=0x7f81e0008040 comp rx=0 tx=0).stop 2023-12-04T05:45:16.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:16.191+0000 7f81ec4eb640 1 -- 172.21.15.107:0/277521321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81e415b400 msgr2=0x7f81e415b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:16.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:16.191+0000 7f81ec4eb640 1 --2- 172.21.15.107:0/277521321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81e415b400 0x7f81e415b880 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f81dc01dbe0 tx=0x7f81d0005800 comp rx=0 tx=0).stop 2023-12-04T05:45:16.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:16.191+0000 7f81ec4eb640 1 -- 172.21.15.107:0/277521321 shutdown_connections 2023-12-04T05:45:16.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:16.191+0000 7f81ec4eb640 1 --2- 172.21.15.107:0/277521321 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f81c80668d0 0x7f81c8068d90 secure :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f81e0009ac0 tx=0x7f81e0008040 comp rx=0 tx=0).stop 2023-12-04T05:45:16.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:16.191+0000 7f81ec4eb640 1 --2- 172.21.15.107:0/277521321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81e415b400 0x7f81e415b880 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:16.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:16.191+0000 7f81ec4eb640 1 --2- 172.21.15.107:0/277521321 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81e40b9970 0x7f81e40b9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:16.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:16.191+0000 7f81ec4eb640 1 --2- 172.21.15.107:0/277521321 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f81d0004d10 0x7f81e41528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:16.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:16.191+0000 7f81ec4eb640 1 -- 172.21.15.107:0/277521321 >> 172.21.15.107:0/277521321 conn(0x7f81e40c7ff0 msgr2=0x7f81e41564e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:16.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:16.191+0000 7f81ec4eb640 1 -- 172.21.15.107:0/277521321 shutdown_connections 2023-12-04T05:45:16.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:16.191+0000 7f81ec4eb640 1 -- 172.21.15.107:0/277521321 wait complete. 2023-12-04T05:45:16.210 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:45:18.211 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:18.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.339+0000 7f913f21d640 1 Processor -- start 2023-12-04T05:45:18.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.339+0000 7f913f21d640 1 -- start start 2023-12-04T05:45:18.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.343+0000 7f913f21d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9138151180 0x7f91380c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:18.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.343+0000 7f913f21d640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9138070380 0x7f9138070780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:18.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.343+0000 7f913f21d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91380700c0 con 0x7f91380714f0 2023-12-04T05:45:18.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.343+0000 7f913f21d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9138156df0 con 0x7f9138151180 2023-12-04T05:45:18.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.343+0000 7f913f21d640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91381570d0 con 0x7f9138070380 2023-12-04T05:45:18.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.343+0000 7f913cfb9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9138151180 0x7f91380c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:18.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.343+0000 7f913cfb9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9138151180 0x7f91380c79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57808/0 (socket says 172.21.15.107:57808) 2023-12-04T05:45:18.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.343+0000 7f913cfb9640 1 -- 172.21.15.107:0/1732335009 learned_addr learned my addr 172.21.15.107:0/1732335009 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:18.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.343+0000 7f913d7ba640 1 --2- 172.21.15.107:0/1732335009 >> v2:172.21.15.195:3300/0 conn(0x7f9138070380 0x7f9138070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:18.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.343+0000 7f913cfb9640 1 -- 172.21.15.107:0/1732335009 >> v2:172.21.15.195:3300/0 conn(0x7f9138070380 msgr2=0x7f9138070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:18.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.343+0000 7f913cfb9640 1 --2- 172.21.15.107:0/1732335009 >> v2:172.21.15.195:3300/0 conn(0x7f9138070380 0x7f9138070780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:18.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.343+0000 7f913cfb9640 1 -- 172.21.15.107:0/1732335009 >> v1:172.21.15.195:6789/0 conn(0x7f91380714f0 legacy=0x7f9138071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:18.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.343+0000 7f913cfb9640 1 -- 172.21.15.107:0/1732335009 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91381573b0 con 0x7f9138151180 2023-12-04T05:45:18.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.343+0000 7f913d7ba640 1 --2- 172.21.15.107:0/1732335009 >> v2:172.21.15.195:3300/0 conn(0x7f9138070380 0x7f9138070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:45:18.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.347+0000 7f913cfb9640 1 --2- 172.21.15.107:0/1732335009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9138151180 0x7f91380c79e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f913806f680 tx=0x7f913401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49038ba16b7b52ad server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:18.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.347+0000 7f912f7fe640 1 -- 172.21.15.107:0/1732335009 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91340a89b0 con 0x7f9138151180 2023-12-04T05:45:18.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.347+0000 7f912f7fe640 1 -- 172.21.15.107:0/1732335009 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91340a8b10 con 0x7f9138151180 2023-12-04T05:45:18.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.347+0000 7f912f7fe640 1 -- 172.21.15.107:0/1732335009 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91340a8ea0 con 0x7f9138151180 2023-12-04T05:45:18.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.347+0000 7f913f21d640 1 -- 172.21.15.107:0/1732335009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9138151180 msgr2=0x7f91380c79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:18.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.347+0000 7f913f21d640 1 --2- 172.21.15.107:0/1732335009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9138151180 0x7f91380c79e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f913806f680 tx=0x7f913401ecb0 comp rx=0 tx=0).stop 2023-12-04T05:45:18.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.347+0000 7f913f21d640 1 -- 172.21.15.107:0/1732335009 shutdown_connections 2023-12-04T05:45:18.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.347+0000 7f913f21d640 1 --2- 172.21.15.107:0/1732335009 >> v2:172.21.15.195:3300/0 conn(0x7f9138070380 0x7f9138070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:18.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.347+0000 7f913f21d640 1 --2- 172.21.15.107:0/1732335009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9138151180 0x7f91380c79e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:18.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.347+0000 7f913f21d640 1 -- 172.21.15.107:0/1732335009 >> 172.21.15.107:0/1732335009 conn(0x7f91380c7ff0 msgr2=0x7f91380c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:18.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.347+0000 7f913f21d640 1 -- 172.21.15.107:0/1732335009 shutdown_connections 2023-12-04T05:45:18.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.347+0000 7f913f21d640 1 -- 172.21.15.107:0/1732335009 wait complete. 2023-12-04T05:45:18.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.347+0000 7f913f21d640 1 Processor -- start 2023-12-04T05:45:18.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.347+0000 7f913f21d640 1 -- start start 2023-12-04T05:45:18.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.347+0000 7f913f21d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9138070380 0x7f91381f3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:18.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.347+0000 7f913f21d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f91380714f0 0x7f91381f3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:18.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.347+0000 7f913f21d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9138151180 0x7f91380ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:18.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.347+0000 7f913f21d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9138158f30 con 0x7f9138070380 2023-12-04T05:45:18.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.347+0000 7f913f21d640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91381592c0 con 0x7f9138151180 2023-12-04T05:45:18.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.347+0000 7f913f21d640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91381f44c0 con 0x7f91380714f0 2023-12-04T05:45:18.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.351+0000 7f913cfb9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9138070380 0x7f91381f3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:18.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.351+0000 7f913cfb9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9138070380 0x7f91381f3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57816/0 (socket says 172.21.15.107:57816) 2023-12-04T05:45:18.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.351+0000 7f913cfb9640 1 -- 172.21.15.107:0/196599676 learned_addr learned my addr 172.21.15.107:0/196599676 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:18.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.351+0000 7f913cfb9640 1 -- 172.21.15.107:0/196599676 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f91380714f0 msgr2=0x7f91381f3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:18.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.351+0000 7f913cfb9640 1 --2- 172.21.15.107:0/196599676 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f91380714f0 0x7f91381f3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:18.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.351+0000 7f913d7ba640 1 --2- 172.21.15.107:0/196599676 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9138151180 0x7f91380ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:18.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.351+0000 7f913cfb9640 1 -- 172.21.15.107:0/196599676 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9138151180 msgr2=0x7f91380ca5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:18.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.351+0000 7f913cfb9640 1 --2- 172.21.15.107:0/196599676 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9138151180 0x7f91380ca5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:18.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.351+0000 7f913cfb9640 1 -- 172.21.15.107:0/196599676 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91340094d0 con 0x7f9138070380 2023-12-04T05:45:18.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.351+0000 7f913cfb9640 1 --2- 172.21.15.107:0/196599676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9138070380 0x7f91381f3950 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f91340b3040 tx=0x7f913401ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:18.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.351+0000 7f912d7fa640 1 -- 172.21.15.107:0/196599676 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91340ba600 con 0x7f9138070380 2023-12-04T05:45:18.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.351+0000 7f912d7fa640 1 -- 172.21.15.107:0/196599676 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91340ba7a0 con 0x7f9138070380 2023-12-04T05:45:18.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.351+0000 7f912d7fa640 1 -- 172.21.15.107:0/196599676 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91340bab50 con 0x7f9138070380 2023-12-04T05:45:18.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.351+0000 7f913f21d640 1 -- 172.21.15.107:0/196599676 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9138155ab0 con 0x7f9138070380 2023-12-04T05:45:18.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.351+0000 7f913f21d640 1 -- 172.21.15.107:0/196599676 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9138156940 con 0x7f9138070380 2023-12-04T05:45:18.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.351+0000 7f913f21d640 1 -- 172.21.15.107:0/196599676 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91380c3010 con 0x7f9138070380 2023-12-04T05:45:18.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.351+0000 7f912d7fa640 1 -- 172.21.15.107:0/196599676 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f91340bacb0 con 0x7f9138070380 2023-12-04T05:45:18.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.351+0000 7f912d7fa640 1 --2- 172.21.15.107:0/196599676 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9124066900 0x7f9124068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:18.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.351+0000 7f912d7fa640 1 -- 172.21.15.107:0/196599676 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f9134122b30 con 0x7f9138070380 2023-12-04T05:45:18.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.351+0000 7f912ffff640 1 --2- 172.21.15.107:0/196599676 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9124066900 0x7f9124068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:18.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.351+0000 7f912ffff640 1 --2- 172.21.15.107:0/196599676 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9124066900 0x7f9124068dc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f91380cb440 tx=0x7f91280079c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:18.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.355+0000 7f912d7fa640 1 -- 172.21.15.107:0/196599676 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f91340efc80 con 0x7f9138070380 2023-12-04T05:45:18.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.543+0000 7f913f21d640 1 -- 172.21.15.107:0/196599676 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91380c3010 con 0x7f9138070380 2023-12-04T05:45:18.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.543+0000 7f912d7fa640 1 -- 172.21.15.107:0/196599676 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f91340f32a0 con 0x7f9138070380 2023-12-04T05:45:18.545 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:18.546 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:14.058884+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3],"up":{"mds_0":4288,"mds_1":5106,"mds_2":4263,"mds_3":4260},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:stopping","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":36,"state":"up:resolve","state_seq":3,"addr":"172.21.15.107:6839/3319682203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":3319682203},{"type":"v1","addr":"172.21.15.107:6839","nonce":3319682203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:18.546 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 37 2023-12-04T05:45:18.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.547+0000 7f913f21d640 1 -- 172.21.15.107:0/196599676 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9124066900 msgr2=0x7f9124068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:18.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.547+0000 7f913f21d640 1 --2- 172.21.15.107:0/196599676 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9124066900 0x7f9124068dc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f91380cb440 tx=0x7f91280079c0 comp rx=0 tx=0).stop 2023-12-04T05:45:18.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.547+0000 7f913f21d640 1 -- 172.21.15.107:0/196599676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9138070380 msgr2=0x7f91381f3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:18.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.547+0000 7f913f21d640 1 --2- 172.21.15.107:0/196599676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9138070380 0x7f91381f3950 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f91340b3040 tx=0x7f913401ba50 comp rx=0 tx=0).stop 2023-12-04T05:45:18.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.547+0000 7f913f21d640 1 -- 172.21.15.107:0/196599676 shutdown_connections 2023-12-04T05:45:18.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.547+0000 7f913f21d640 1 --2- 172.21.15.107:0/196599676 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9124066900 0x7f9124068dc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:18.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.547+0000 7f913f21d640 1 --2- 172.21.15.107:0/196599676 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9138151180 0x7f91380ca5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:18.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.547+0000 7f913f21d640 1 --2- 172.21.15.107:0/196599676 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f91380714f0 0x7f91381f3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:18.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.547+0000 7f913f21d640 1 --2- 172.21.15.107:0/196599676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9138070380 0x7f91381f3950 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:18.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.547+0000 7f913f21d640 1 -- 172.21.15.107:0/196599676 >> 172.21.15.107:0/196599676 conn(0x7f91380c7ff0 msgr2=0x7f91380c5ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:18.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.547+0000 7f913f21d640 1 -- 172.21.15.107:0/196599676 shutdown_connections 2023-12-04T05:45:18.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:18.547+0000 7f913f21d640 1 -- 172.21.15.107:0/196599676 wait complete. 2023-12-04T05:45:18.565 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-12-04T05:45:18.566 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 37, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-12-04T05:39:59.032019+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 37, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3 ], "info": { "gid_4260": { "addr": "172.21.15.107:6836/3444851515", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 3444851515, "type": "v2" }, { "addr": "172.21.15.107:6836", "nonce": 3444851515, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4260, "incarnation": 14, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:stopping", "state_seq": 9 }, "gid_4263": { "addr": "172.21.15.107:6837/2210527429", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6835", "nonce": 2210527429, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 2210527429, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4263, "incarnation": 12, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 7 }, "gid_4288": { "addr": "172.21.15.195:6837/3412865477", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6835", "nonce": 3412865477, "type": "v2" }, { "addr": "172.21.15.195:6837", "nonce": 3412865477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_5106": { "addr": "172.21.15.107:6839/3319682203", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6838", "nonce": 3319682203, "type": "v2" }, { "addr": "172.21.15.107:6839", "nonce": 3319682203, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5106, "incarnation": 36, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:resolve", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 26, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-12-04T05:45:14.058884+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 4 ], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 5106, "mds_2": 4263, "mds_3": 4260 } } } ], "standbys": [] } 2023-12-04T05:45:20.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:20.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.699+0000 7f561cf42640 1 Processor -- start 2023-12-04T05:45:20.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.699+0000 7f561cf42640 1 -- start start 2023-12-04T05:45:20.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.699+0000 7f561cf42640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f56180c46a0 0x7f56180c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:20.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.699+0000 7f561cf42640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f561806e660 con 0x7f56180c4a80 2023-12-04T05:45:20.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.699+0000 7f561cf42640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5618156cf0 con 0x7f5618157180 2023-12-04T05:45:20.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.699+0000 7f561cf42640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5618156490 con 0x7f56180c46a0 2023-12-04T05:45:20.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.699+0000 7f561659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f56180c46a0 0x7f56180c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:20.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.699+0000 7f5616d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f56180c4a80 0x7f56180c50e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59186/0 (socket says 172.21.15.107:59186) 2023-12-04T05:45:20.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.699+0000 7f5616d9d640 1 -- 172.21.15.107:0/1070846684 learned_addr learned my addr 172.21.15.107:0/1070846684 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:45:20.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.699+0000 7f561659c640 1 -- 172.21.15.107:0/1070846684 >> v1:172.21.15.195:6789/0 conn(0x7f5618157180 legacy=0x7f56180c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:20.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.699+0000 7f561659c640 1 -- 172.21.15.107:0/1070846684 >> v1:172.21.15.107:6789/0 conn(0x7f56180c4a80 legacy=0x7f56180c50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:20.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.699+0000 7f5615d9b640 1 -- 172.21.15.107:0/1070846684 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193572563 0 0) 0x7f56100014a0 con 0x7f56180c4a80 2023-12-04T05:45:20.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.699+0000 7f561659c640 1 -- 172.21.15.107:0/1070846684 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561806e0a0 con 0x7f56180c46a0 2023-12-04T05:45:20.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561659c640 1 --2- 172.21.15.107:0/1070846684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56180c46a0 0x7f56180c8b10 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f56080097c0 tx=0x7f560801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8099c11b5a44f242 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:20.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f5615d9b640 1 -- 172.21.15.107:0/1070846684 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56080ab600 con 0x7f56180c46a0 2023-12-04T05:45:20.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f5615d9b640 1 -- 172.21.15.107:0/1070846684 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56080a7050 con 0x7f56180c46a0 2023-12-04T05:45:20.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f5615d9b640 1 -- 172.21.15.107:0/1070846684 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56080ab980 con 0x7f56180c46a0 2023-12-04T05:45:20.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561cf42640 1 -- 172.21.15.107:0/1070846684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56180c46a0 msgr2=0x7f56180c8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:20.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561cf42640 1 --2- 172.21.15.107:0/1070846684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56180c46a0 0x7f56180c8b10 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f56080097c0 tx=0x7f560801c440 comp rx=0 tx=0).stop 2023-12-04T05:45:20.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561cf42640 1 -- 172.21.15.107:0/1070846684 shutdown_connections 2023-12-04T05:45:20.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561cf42640 1 --2- 172.21.15.107:0/1070846684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56180c46a0 0x7f56180c8b10 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:20.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561cf42640 1 -- 172.21.15.107:0/1070846684 >> 172.21.15.107:0/1070846684 conn(0x7f56180c7ff0 msgr2=0x7f56180c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:20.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561cf42640 1 -- 172.21.15.107:0/1070846684 shutdown_connections 2023-12-04T05:45:20.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561cf42640 1 -- 172.21.15.107:0/1070846684 wait complete. 2023-12-04T05:45:20.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561cf42640 1 Processor -- start 2023-12-04T05:45:20.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561cf42640 1 -- start start 2023-12-04T05:45:20.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561cf42640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56180c46a0 0x7f5618152300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:20.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561cf42640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f56180c4a80 0x7f56180c93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:20.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561cf42640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5618157180 0x7f56181644f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:20.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561cf42640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5618152f20 con 0x7f56180c46a0 2023-12-04T05:45:20.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561cf42640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56181f2880 con 0x7f56180c4a80 2023-12-04T05:45:20.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561cf42640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5618152960 con 0x7f5618157180 2023-12-04T05:45:20.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56180c46a0 0x7f5618152300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:20.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56180c46a0 0x7f5618152300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57822/0 (socket says 172.21.15.107:57822) 2023-12-04T05:45:20.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561659c640 1 -- 172.21.15.107:0/802395217 learned_addr learned my addr 172.21.15.107:0/802395217 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:20.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561659c640 1 -- 172.21.15.107:0/802395217 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5618157180 msgr2=0x7f56181644f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:20.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561659c640 1 --2- 172.21.15.107:0/802395217 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5618157180 0x7f56181644f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:20.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f560ffff640 1 --2- 172.21.15.107:0/802395217 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f56180c4a80 0x7f56180c93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:20.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561659c640 1 -- 172.21.15.107:0/802395217 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f56180c4a80 msgr2=0x7f56180c93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561659c640 1 --2- 172.21.15.107:0/802395217 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f56180c4a80 0x7f56180c93f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561659c640 1 -- 172.21.15.107:0/802395217 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5608009420 con 0x7f56180c46a0 2023-12-04T05:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f561659c640 1 --2- 172.21.15.107:0/802395217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56180c46a0 0x7f5618152300 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f56080ae040 tx=0x7f560801ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.703+0000 7f560effd640 1 -- 172.21.15.107:0/802395217 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56080b7650 con 0x7f56180c46a0 2023-12-04T05:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.707+0000 7f560effd640 1 -- 172.21.15.107:0/802395217 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56080b77b0 con 0x7f56180c46a0 2023-12-04T05:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.707+0000 7f561cf42640 1 -- 172.21.15.107:0/802395217 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5618152c40 con 0x7f56180c46a0 2023-12-04T05:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.707+0000 7f560effd640 1 -- 172.21.15.107:0/802395217 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56080b7b60 con 0x7f56180c46a0 2023-12-04T05:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.707+0000 7f561cf42640 1 -- 172.21.15.107:0/802395217 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56181f2e00 con 0x7f56180c46a0 2023-12-04T05:45:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.707+0000 7f561cf42640 1 -- 172.21.15.107:0/802395217 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56181f26e0 con 0x7f56180c46a0 2023-12-04T05:45:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.707+0000 7f560effd640 1 -- 172.21.15.107:0/802395217 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f56080b7cc0 con 0x7f56180c46a0 2023-12-04T05:45:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.707+0000 7f560effd640 1 --2- 172.21.15.107:0/802395217 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f55fc066900 0x7f55fc068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.707+0000 7f560effd640 1 -- 172.21.15.107:0/802395217 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f560811f050 con 0x7f56180c46a0 2023-12-04T05:45:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.707+0000 7f560ffff640 1 --2- 172.21.15.107:0/802395217 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f55fc066900 0x7f55fc068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.707+0000 7f560ffff640 1 --2- 172.21.15.107:0/802395217 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f55fc066900 0x7f55fc068dc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5618150e70 tx=0x7f5600008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:20.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.711+0000 7f560effd640 1 -- 172.21.15.107:0/802395217 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f56181f26e0 con 0x7f56180c46a0 2023-12-04T05:45:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.899+0000 7f561cf42640 1 -- 172.21.15.107:0/802395217 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56181f26e0 con 0x7f56180c46a0 2023-12-04T05:45:20.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.899+0000 7f560effd640 1 -- 172.21.15.107:0/802395217 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+4976 (secure 0 0 0) 0x7f56181f26e0 con 0x7f56180c46a0 2023-12-04T05:45:20.902 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:20.902 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:20.064732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3],"up":{"mds_0":4288,"mds_1":5106,"mds_2":4263,"mds_3":4260},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:stopping","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":36,"state":"up:reconnect","state_seq":5,"addr":"172.21.15.107:6839/3319682203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":3319682203},{"type":"v1","addr":"172.21.15.107:6839","nonce":3319682203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:20.902 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 38 2023-12-04T05:45:20.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.903+0000 7f561cf42640 1 -- 172.21.15.107:0/802395217 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f55fc066900 msgr2=0x7f55fc068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:20.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.903+0000 7f561cf42640 1 --2- 172.21.15.107:0/802395217 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f55fc066900 0x7f55fc068dc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5618150e70 tx=0x7f5600008040 comp rx=0 tx=0).stop 2023-12-04T05:45:20.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.903+0000 7f561cf42640 1 -- 172.21.15.107:0/802395217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56180c46a0 msgr2=0x7f5618152300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:20.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.903+0000 7f561cf42640 1 --2- 172.21.15.107:0/802395217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56180c46a0 0x7f5618152300 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f56080ae040 tx=0x7f560801ac80 comp rx=0 tx=0).stop 2023-12-04T05:45:20.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.903+0000 7f561cf42640 1 -- 172.21.15.107:0/802395217 shutdown_connections 2023-12-04T05:45:20.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.903+0000 7f561cf42640 1 --2- 172.21.15.107:0/802395217 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f55fc066900 0x7f55fc068dc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:20.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.903+0000 7f561cf42640 1 --2- 172.21.15.107:0/802395217 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5618157180 0x7f56181644f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:20.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.903+0000 7f561cf42640 1 --2- 172.21.15.107:0/802395217 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f56180c4a80 0x7f56180c93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:20.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.903+0000 7f561cf42640 1 --2- 172.21.15.107:0/802395217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56180c46a0 0x7f5618152300 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:20.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.903+0000 7f561cf42640 1 -- 172.21.15.107:0/802395217 >> 172.21.15.107:0/802395217 conn(0x7f56180c7ff0 msgr2=0x7f561814d270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:20.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.903+0000 7f561cf42640 1 -- 172.21.15.107:0/802395217 shutdown_connections 2023-12-04T05:45:20.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:20.903+0000 7f561cf42640 1 -- 172.21.15.107:0/802395217 wait complete. 2023-12-04T05:45:20.921 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:45:21.706 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~94s 2023-12-04T05:45:21.706 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~135s 2023-12-04T05:45:22.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:23.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.047+0000 7fbe8d75b640 1 Processor -- start 2023-12-04T05:45:23.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.047+0000 7fbe8d75b640 1 -- start start 2023-12-04T05:45:23.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.047+0000 7fbe8d75b640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbe88151180 0x7fbe880c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:23.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.047+0000 7fbe8d75b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbe88070380 0x7fbe88070790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.047+0000 7fbe8d75b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe880700c0 con 0x7fbe880714f0 2023-12-04T05:45:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.047+0000 7fbe8d75b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe88156df0 con 0x7fbe88070380 2023-12-04T05:45:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.047+0000 7fbe8d75b640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe881570d0 con 0x7fbe88151180 2023-12-04T05:45:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.047+0000 7fbe867fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbe880714f0 0x7fbe880718f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59196/0 (socket says 172.21.15.107:59196) 2023-12-04T05:45:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.047+0000 7fbe867fc640 1 -- 172.21.15.107:0/3031467526 learned_addr learned my addr 172.21.15.107:0/3031467526 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:45:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.047+0000 7fbe877fe640 1 --2- 172.21.15.107:0/3031467526 >> v2:172.21.15.107:3300/0 conn(0x7fbe88070380 0x7fbe88070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.047+0000 7fbe86ffd640 1 --2- 172.21.15.107:0/3031467526 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbe88151180 0x7fbe880c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.047+0000 7fbe85ffb640 1 -- 172.21.15.107:0/3031467526 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 293420173 0 0) 0x7fbe880700c0 con 0x7fbe880714f0 2023-12-04T05:45:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.047+0000 7fbe85ffb640 1 -- 172.21.15.107:0/3031467526 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe70003530 con 0x7fbe880714f0 2023-12-04T05:45:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.047+0000 7fbe877fe640 1 -- 172.21.15.107:0/3031467526 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbe88151180 msgr2=0x7fbe880c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.047+0000 7fbe877fe640 1 --2- 172.21.15.107:0/3031467526 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbe88151180 0x7fbe880c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.047+0000 7fbe877fe640 1 -- 172.21.15.107:0/3031467526 >> v1:172.21.15.107:6789/0 conn(0x7fbe880714f0 legacy=0x7fbe880718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.047+0000 7fbe877fe640 1 -- 172.21.15.107:0/3031467526 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe881573b0 con 0x7fbe88070380 2023-12-04T05:45:23.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.051+0000 7fbe877fe640 1 --2- 172.21.15.107:0/3031467526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe88070380 0x7fbe88070790 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fbe80009d90 tx=0x7fbe8001c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c9583b125c09e8 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:23.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.051+0000 7fbe85ffb640 1 -- 172.21.15.107:0/3031467526 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe800a59e0 con 0x7fbe88070380 2023-12-04T05:45:23.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.051+0000 7fbe85ffb640 1 -- 172.21.15.107:0/3031467526 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe800a5b40 con 0x7fbe88070380 2023-12-04T05:45:23.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.051+0000 7fbe85ffb640 1 -- 172.21.15.107:0/3031467526 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe800ad440 con 0x7fbe88070380 2023-12-04T05:45:23.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.051+0000 7fbe8d75b640 1 -- 172.21.15.107:0/3031467526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe88070380 msgr2=0x7fbe88070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:23.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.051+0000 7fbe8d75b640 1 --2- 172.21.15.107:0/3031467526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe88070380 0x7fbe88070790 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fbe80009d90 tx=0x7fbe8001c920 comp rx=0 tx=0).stop 2023-12-04T05:45:23.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.051+0000 7fbe8d75b640 1 -- 172.21.15.107:0/3031467526 shutdown_connections 2023-12-04T05:45:23.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.051+0000 7fbe8d75b640 1 --2- 172.21.15.107:0/3031467526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe88070380 0x7fbe88070790 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:23.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.051+0000 7fbe8d75b640 1 --2- 172.21.15.107:0/3031467526 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbe88151180 0x7fbe880c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:23.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.051+0000 7fbe8d75b640 1 -- 172.21.15.107:0/3031467526 >> 172.21.15.107:0/3031467526 conn(0x7fbe880c7ff0 msgr2=0x7fbe880c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:23.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.051+0000 7fbe8d75b640 1 -- 172.21.15.107:0/3031467526 shutdown_connections 2023-12-04T05:45:23.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.051+0000 7fbe8d75b640 1 -- 172.21.15.107:0/3031467526 wait complete. 2023-12-04T05:45:23.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.051+0000 7fbe8d75b640 1 Processor -- start 2023-12-04T05:45:23.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.051+0000 7fbe8d75b640 1 -- start start 2023-12-04T05:45:23.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe8d75b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe88070380 0x7fbe880c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:23.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe8d75b640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbe880714f0 0x7fbe8813bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:23.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe8d75b640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbe88151180 0x7fbe8813c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:23.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe8d75b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8814c880 con 0x7fbe88070380 2023-12-04T05:45:23.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe8d75b640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8813dec0 con 0x7fbe88151180 2023-12-04T05:45:23.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe8d75b640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8813e140 con 0x7fbe880714f0 2023-12-04T05:45:23.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe86ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe88070380 0x7fbe880c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:23.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe86ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe88070380 0x7fbe880c4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57828/0 (socket says 172.21.15.107:57828) 2023-12-04T05:45:23.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe86ffd640 1 -- 172.21.15.107:0/2996495010 learned_addr learned my addr 172.21.15.107:0/2996495010 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:23.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe867fc640 1 --2- 172.21.15.107:0/2996495010 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbe880714f0 0x7fbe8813bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:23.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe877fe640 1 --2- 172.21.15.107:0/2996495010 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbe88151180 0x7fbe8813c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:23.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe86ffd640 1 -- 172.21.15.107:0/2996495010 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbe880714f0 msgr2=0x7fbe8813bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:23.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe86ffd640 1 --2- 172.21.15.107:0/2996495010 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbe880714f0 0x7fbe8813bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:23.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe86ffd640 1 -- 172.21.15.107:0/2996495010 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbe88151180 msgr2=0x7fbe8813c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:23.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe86ffd640 1 --2- 172.21.15.107:0/2996495010 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbe88151180 0x7fbe8813c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:23.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe86ffd640 1 -- 172.21.15.107:0/2996495010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe80009a70 con 0x7fbe88070380 2023-12-04T05:45:23.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe867fc640 1 --2- 172.21.15.107:0/2996495010 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbe880714f0 0x7fbe8813bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:45:23.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe86ffd640 1 --2- 172.21.15.107:0/2996495010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe88070380 0x7fbe880c4ba0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbe7800e870 tx=0x7fbe7800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:23.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe6f7fe640 1 -- 172.21.15.107:0/2996495010 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe78009510 con 0x7fbe88070380 2023-12-04T05:45:23.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe6f7fe640 1 -- 172.21.15.107:0/2996495010 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe7800eec0 con 0x7fbe88070380 2023-12-04T05:45:23.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe8d75b640 1 -- 172.21.15.107:0/2996495010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe880cafa0 con 0x7fbe88070380 2023-12-04T05:45:23.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe6f7fe640 1 -- 172.21.15.107:0/2996495010 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7800cb40 con 0x7fbe88070380 2023-12-04T05:45:23.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe8d75b640 1 -- 172.21.15.107:0/2996495010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe881f0b60 con 0x7fbe88070380 2023-12-04T05:45:23.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe8d75b640 1 -- 172.21.15.107:0/2996495010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe880cae00 con 0x7fbe88070380 2023-12-04T05:45:23.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe6f7fe640 1 -- 172.21.15.107:0/2996495010 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fbe7800cca0 con 0x7fbe88070380 2023-12-04T05:45:23.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe6f7fe640 1 --2- 172.21.15.107:0/2996495010 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbe74066900 0x7fbe74068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:23.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe6f7fe640 1 -- 172.21.15.107:0/2996495010 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6083+0+0 (secure 0 0 0) 0x7fbe780846e0 con 0x7fbe88070380 2023-12-04T05:45:23.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe867fc640 1 --2- 172.21.15.107:0/2996495010 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbe74066900 0x7fbe74068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:23.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.055+0000 7fbe867fc640 1 --2- 172.21.15.107:0/2996495010 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbe74066900 0x7fbe74068dc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbe881565d0 tx=0x7fbe7c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:23.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.059+0000 7fbe6f7fe640 1 -- 172.21.15.107:0/2996495010 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbe880cae00 con 0x7fbe88070380 2023-12-04T05:45:23.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.235+0000 7fbe8d75b640 1 -- 172.21.15.107:0/2996495010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe880cae00 con 0x7fbe88070380 2023-12-04T05:45:23.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.235+0000 7fbe6f7fe640 1 -- 172.21.15.107:0/2996495010 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+4976 (secure 0 0 0) 0x7fbe880cae00 con 0x7fbe88070380 2023-12-04T05:45:23.237 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:23.237 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:20.064732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3],"up":{"mds_0":4288,"mds_1":5106,"mds_2":4263,"mds_3":4260},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:stopping","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":36,"state":"up:reconnect","state_seq":5,"addr":"172.21.15.107:6839/3319682203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":3319682203},{"type":"v1","addr":"172.21.15.107:6839","nonce":3319682203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:23.237 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 38 2023-12-04T05:45:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.235+0000 7fbe8d75b640 1 -- 172.21.15.107:0/2996495010 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbe74066900 msgr2=0x7fbe74068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.235+0000 7fbe8d75b640 1 --2- 172.21.15.107:0/2996495010 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbe74066900 0x7fbe74068dc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbe881565d0 tx=0x7fbe7c009040 comp rx=0 tx=0).stop 2023-12-04T05:45:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.235+0000 7fbe8d75b640 1 -- 172.21.15.107:0/2996495010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe88070380 msgr2=0x7fbe880c4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.235+0000 7fbe8d75b640 1 --2- 172.21.15.107:0/2996495010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe88070380 0x7fbe880c4ba0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbe7800e870 tx=0x7fbe7800ec70 comp rx=0 tx=0).stop 2023-12-04T05:45:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.235+0000 7fbe8d75b640 1 -- 172.21.15.107:0/2996495010 shutdown_connections 2023-12-04T05:45:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.235+0000 7fbe8d75b640 1 --2- 172.21.15.107:0/2996495010 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbe74066900 0x7fbe74068dc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.235+0000 7fbe8d75b640 1 --2- 172.21.15.107:0/2996495010 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbe88151180 0x7fbe8813c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.235+0000 7fbe8d75b640 1 --2- 172.21.15.107:0/2996495010 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbe880714f0 0x7fbe8813bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.235+0000 7fbe8d75b640 1 --2- 172.21.15.107:0/2996495010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe88070380 0x7fbe880c4ba0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.235+0000 7fbe8d75b640 1 -- 172.21.15.107:0/2996495010 >> 172.21.15.107:0/2996495010 conn(0x7fbe880c7ff0 msgr2=0x7fbe880c6d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:23.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.239+0000 7fbe8d75b640 1 -- 172.21.15.107:0/2996495010 shutdown_connections 2023-12-04T05:45:23.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:23.239+0000 7fbe8d75b640 1 -- 172.21.15.107:0/2996495010 wait complete. 2023-12-04T05:45:23.254 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:45:25.256 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:25.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.383+0000 7fc55eec5640 1 Processor -- start 2023-12-04T05:45:25.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.383+0000 7fc55eec5640 1 -- start start 2023-12-04T05:45:25.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.387+0000 7fc55eec5640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc558072110 0x7fc558158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:25.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.387+0000 7fc55eec5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc5580724f0 0x7fc5580c7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:25.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.387+0000 7fc55eec5640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc558158da0 con 0x7fc5580c56f0 2023-12-04T05:45:25.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.387+0000 7fc55eec5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5581583d0 con 0x7fc5580724f0 2023-12-04T05:45:25.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.387+0000 7fc55eec5640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5580c6170 con 0x7fc558072110 2023-12-04T05:45:25.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.387+0000 7fc54ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc5580724f0 0x7fc5580c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:25.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.387+0000 7fc54ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc5580724f0 0x7fc5580c7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57838/0 (socket says 172.21.15.107:57838) 2023-12-04T05:45:25.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.387+0000 7fc54ffff640 1 -- 172.21.15.107:0/2499766318 learned_addr learned my addr 172.21.15.107:0/2499766318 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:25.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.387+0000 7fc55cc61640 1 --2- 172.21.15.107:0/2499766318 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc558072110 0x7fc558158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:25.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.387+0000 7fc54ffff640 1 -- 172.21.15.107:0/2499766318 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc558072110 msgr2=0x7fc558158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:25.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.387+0000 7fc54ffff640 1 --2- 172.21.15.107:0/2499766318 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc558072110 0x7fc558158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:25.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.387+0000 7fc54ffff640 1 -- 172.21.15.107:0/2499766318 >> v1:172.21.15.195:6789/0 conn(0x7fc5580c56f0 legacy=0x7fc558140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:25.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.387+0000 7fc54ffff640 1 -- 172.21.15.107:0/2499766318 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc558157da0 con 0x7fc5580724f0 2023-12-04T05:45:25.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.387+0000 7fc55cc61640 1 --2- 172.21.15.107:0/2499766318 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc558072110 0x7fc558158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:45:25.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.387+0000 7fc54ffff640 1 --2- 172.21.15.107:0/2499766318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5580724f0 0x7fc5580c7420 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc548009870 tx=0x7fc54801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b241f8bf4967205 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:25.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.387+0000 7fc54f7fe640 1 -- 172.21.15.107:0/2499766318 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5480ac410 con 0x7fc5580724f0 2023-12-04T05:45:25.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.387+0000 7fc54f7fe640 1 -- 172.21.15.107:0/2499766318 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5480ac570 con 0x7fc5580724f0 2023-12-04T05:45:25.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.387+0000 7fc54f7fe640 1 -- 172.21.15.107:0/2499766318 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5480ac900 con 0x7fc5580724f0 2023-12-04T05:45:25.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55eec5640 1 -- 172.21.15.107:0/2499766318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5580724f0 msgr2=0x7fc5580c7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:25.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55eec5640 1 --2- 172.21.15.107:0/2499766318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5580724f0 0x7fc5580c7420 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc548009870 tx=0x7fc54801c500 comp rx=0 tx=0).stop 2023-12-04T05:45:25.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55eec5640 1 -- 172.21.15.107:0/2499766318 shutdown_connections 2023-12-04T05:45:25.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55eec5640 1 --2- 172.21.15.107:0/2499766318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5580724f0 0x7fc5580c7420 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:25.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55eec5640 1 --2- 172.21.15.107:0/2499766318 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc558072110 0x7fc558158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:25.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55eec5640 1 -- 172.21.15.107:0/2499766318 >> 172.21.15.107:0/2499766318 conn(0x7fc5580c7ff0 msgr2=0x7fc5580c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:25.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55eec5640 1 -- 172.21.15.107:0/2499766318 shutdown_connections 2023-12-04T05:45:25.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55eec5640 1 -- 172.21.15.107:0/2499766318 wait complete. 2023-12-04T05:45:25.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55eec5640 1 Processor -- start 2023-12-04T05:45:25.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55eec5640 1 -- start start 2023-12-04T05:45:25.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55eec5640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc558072110 0x7fc558155340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:25.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55eec5640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5580724f0 0x7fc558154240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:25.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55eec5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5580c56f0 0x7fc558156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:25.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55eec5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5581548d0 con 0x7fc5580c56f0 2023-12-04T05:45:25.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55eec5640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc558136cf0 con 0x7fc5580724f0 2023-12-04T05:45:25.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55eec5640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5581360e0 con 0x7fc558072110 2023-12-04T05:45:25.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55d462640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5580c56f0 0x7fc558156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:25.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55d462640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5580c56f0 0x7fc558156980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57842/0 (socket says 172.21.15.107:57842) 2023-12-04T05:45:25.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55d462640 1 -- 172.21.15.107:0/3893437985 learned_addr learned my addr 172.21.15.107:0/3893437985 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:25.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55cc61640 1 --2- 172.21.15.107:0/3893437985 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc558072110 0x7fc558155340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:25.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55d462640 1 -- 172.21.15.107:0/3893437985 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc558072110 msgr2=0x7fc558155340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:25.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55d462640 1 --2- 172.21.15.107:0/3893437985 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc558072110 0x7fc558155340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:25.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55d462640 1 -- 172.21.15.107:0/3893437985 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5580724f0 msgr2=0x7fc558154240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:25.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55d462640 1 --2- 172.21.15.107:0/3893437985 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5580724f0 0x7fc558154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:25.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55d462640 1 -- 172.21.15.107:0/3893437985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5480094d0 con 0x7fc5580c56f0 2023-12-04T05:45:25.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55cc61640 1 --2- 172.21.15.107:0/3893437985 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc558072110 0x7fc558155340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:45:25.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55d462640 1 --2- 172.21.15.107:0/3893437985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5580c56f0 0x7fc558156980 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc55000c780 tx=0x7fc55000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:25.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc54d7fa640 1 -- 172.21.15.107:0/3893437985 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc550003d90 con 0x7fc5580c56f0 2023-12-04T05:45:25.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc54d7fa640 1 -- 172.21.15.107:0/3893437985 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55000ce90 con 0x7fc5580c56f0 2023-12-04T05:45:25.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55eec5640 1 -- 172.21.15.107:0/3893437985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5581363c0 con 0x7fc5580c56f0 2023-12-04T05:45:25.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc54d7fa640 1 -- 172.21.15.107:0/3893437985 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5500052b0 con 0x7fc5580c56f0 2023-12-04T05:45:25.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.391+0000 7fc55eec5640 1 -- 172.21.15.107:0/3893437985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc558136980 con 0x7fc5580c56f0 2023-12-04T05:45:25.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.395+0000 7fc55eec5640 1 -- 172.21.15.107:0/3893437985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc558136e60 con 0x7fc5580c56f0 2023-12-04T05:45:25.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.395+0000 7fc54d7fa640 1 -- 172.21.15.107:0/3893437985 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fc550005410 con 0x7fc5580c56f0 2023-12-04T05:45:25.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.395+0000 7fc54d7fa640 1 --2- 172.21.15.107:0/3893437985 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc5440668d0 0x7fc544068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:25.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.395+0000 7fc54d7fa640 1 -- 172.21.15.107:0/3893437985 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6083+0+0 (secure 0 0 0) 0x7fc550083e20 con 0x7fc5580c56f0 2023-12-04T05:45:25.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.395+0000 7fc55cc61640 1 --2- 172.21.15.107:0/3893437985 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc5440668d0 0x7fc544068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:25.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.395+0000 7fc55cc61640 1 --2- 172.21.15.107:0/3893437985 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc5440668d0 0x7fc544068d90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc554008890 tx=0x7fc554008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:25.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.399+0000 7fc54d7fa640 1 -- 172.21.15.107:0/3893437985 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc550050ee0 con 0x7fc5580c56f0 2023-12-04T05:45:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.579+0000 7fc55eec5640 1 -- 172.21.15.107:0/3893437985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc558067870 con 0x7fc5580c56f0 2023-12-04T05:45:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.583+0000 7fc54d7fa640 1 -- 172.21.15.107:0/3893437985 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+4976 (secure 0 0 0) 0x7fc550017070 con 0x7fc5580c56f0 2023-12-04T05:45:25.583 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:25.583 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:20.064732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3],"up":{"mds_0":4288,"mds_1":5106,"mds_2":4263,"mds_3":4260},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:stopping","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":36,"state":"up:reconnect","state_seq":5,"addr":"172.21.15.107:6839/3319682203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":3319682203},{"type":"v1","addr":"172.21.15.107:6839","nonce":3319682203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 38 2023-12-04T05:45:25.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.583+0000 7fc55eec5640 1 -- 172.21.15.107:0/3893437985 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc5440668d0 msgr2=0x7fc544068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:25.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.583+0000 7fc55eec5640 1 --2- 172.21.15.107:0/3893437985 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc5440668d0 0x7fc544068d90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc554008890 tx=0x7fc554008040 comp rx=0 tx=0).stop 2023-12-04T05:45:25.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.583+0000 7fc55eec5640 1 -- 172.21.15.107:0/3893437985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5580c56f0 msgr2=0x7fc558156980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:25.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.583+0000 7fc55eec5640 1 --2- 172.21.15.107:0/3893437985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5580c56f0 0x7fc558156980 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc55000c780 tx=0x7fc55000cc40 comp rx=0 tx=0).stop 2023-12-04T05:45:25.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.583+0000 7fc55eec5640 1 -- 172.21.15.107:0/3893437985 shutdown_connections 2023-12-04T05:45:25.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.583+0000 7fc55eec5640 1 --2- 172.21.15.107:0/3893437985 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc5440668d0 0x7fc544068d90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:25.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.583+0000 7fc55eec5640 1 --2- 172.21.15.107:0/3893437985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5580c56f0 0x7fc558156980 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:25.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.583+0000 7fc55eec5640 1 --2- 172.21.15.107:0/3893437985 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5580724f0 0x7fc558154240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:25.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.583+0000 7fc55eec5640 1 --2- 172.21.15.107:0/3893437985 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc558072110 0x7fc558155340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:25.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.583+0000 7fc55eec5640 1 -- 172.21.15.107:0/3893437985 >> 172.21.15.107:0/3893437985 conn(0x7fc5580c7ff0 msgr2=0x7fc5581574f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:25.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.583+0000 7fc55eec5640 1 -- 172.21.15.107:0/3893437985 shutdown_connections 2023-12-04T05:45:25.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:25.583+0000 7fc55eec5640 1 -- 172.21.15.107:0/3893437985 wait complete. 2023-12-04T05:45:25.601 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:45:27.602 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:27.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca3091a640 1 Processor -- start 2023-12-04T05:45:27.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca3091a640 1 -- start start 2023-12-04T05:45:27.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca3091a640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fca2c0c8b60 0x7fca2c06e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:27.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca3091a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2c06f340 con 0x7fca2c06e300 2023-12-04T05:45:27.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca3091a640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2c0c6f70 con 0x7fca2c071160 2023-12-04T05:45:27.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca3091a640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca2c0c6d90 con 0x7fca2c0c8b60 2023-12-04T05:45:27.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2a59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fca2c06e300 0x7fca2c1575a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59678/0 (socket says 172.21.15.107:59678) 2023-12-04T05:45:27.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2a59c640 1 -- 172.21.15.107:0/18963391 learned_addr learned my addr 172.21.15.107:0/18963391 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:45:27.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca29d9b640 1 --2- 172.21.15.107:0/18963391 >> v2:172.21.15.195:3300/0 conn(0x7fca2c0c8b60 0x7fca2c06e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:27.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2959a640 1 -- 172.21.15.107:0/18963391 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821677020 0 0) 0x7fca2c06f340 con 0x7fca2c06e300 2023-12-04T05:45:27.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2959a640 1 -- 172.21.15.107:0/18963391 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca14003530 con 0x7fca2c06e300 2023-12-04T05:45:27.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2959a640 1 -- 172.21.15.107:0/18963391 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 85231381 0 0) 0x7fca14003530 con 0x7fca2c06e300 2023-12-04T05:45:27.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2959a640 1 -- 172.21.15.107:0/18963391 >> v2:172.21.15.195:3300/0 conn(0x7fca2c0c8b60 msgr2=0x7fca2c06e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:27.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2959a640 1 --2- 172.21.15.107:0/18963391 >> v2:172.21.15.195:3300/0 conn(0x7fca2c0c8b60 0x7fca2c06e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:27.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2959a640 1 -- 172.21.15.107:0/18963391 >> v1:172.21.15.195:6789/0 conn(0x7fca2c071160 legacy=0x7fca2c1614f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:27.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2959a640 1 -- 172.21.15.107:0/18963391 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2c0c7250 con 0x7fca2c06e300 2023-12-04T05:45:27.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2959a640 1 -- 172.21.15.107:0/18963391 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7fca1c001a00 con 0x7fca2c06e300 2023-12-04T05:45:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2959a640 1 -- 172.21.15.107:0/18963391 >> v1:172.21.15.107:6789/0 conn(0x7fca2c06e300 legacy=0x7fca2c1575a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2959a640 1 --2- 172.21.15.107:0/18963391 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fca14003980 0x7fca14003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2959a640 1 --2- 172.21.15.107:0/18963391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fca140047d0 0x7fca14004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2959a640 1 --2- 172.21.15.107:0/18963391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca14005190 0x7fca1400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2959a640 1 -- 172.21.15.107:0/18963391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c0c6d90 con 0x7fca14005190 2023-12-04T05:45:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2959a640 1 -- 172.21.15.107:0/18963391 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1400c040 con 0x7fca14003980 2023-12-04T05:45:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2959a640 1 -- 172.21.15.107:0/18963391 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1400c200 con 0x7fca140047d0 2023-12-04T05:45:27.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2ad9d640 1 --2- 172.21.15.107:0/18963391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca14005190 0x7fca1400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:27.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2a59c640 1 --2- 172.21.15.107:0/18963391 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fca14003980 0x7fca14003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:27.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca29d9b640 1 --2- 172.21.15.107:0/18963391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fca140047d0 0x7fca14004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:27.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2ad9d640 1 -- 172.21.15.107:0/18963391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fca140047d0 msgr2=0x7fca14004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:27.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2ad9d640 1 --2- 172.21.15.107:0/18963391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fca140047d0 0x7fca14004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:27.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2ad9d640 1 -- 172.21.15.107:0/18963391 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fca14003980 msgr2=0x7fca14003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:27.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2ad9d640 1 --2- 172.21.15.107:0/18963391 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fca14003980 0x7fca14003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:27.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca2ad9d640 1 -- 172.21.15.107:0/18963391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca14003530 con 0x7fca14005190 2023-12-04T05:45:27.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.731+0000 7fca29d9b640 1 --2- 172.21.15.107:0/18963391 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fca140047d0 0x7fca14004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:45:27.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.735+0000 7fca2ad9d640 1 -- 172.21.15.107:0/18963391 reap_dead start 2023-12-04T05:45:27.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.735+0000 7fca2ad9d640 1 --2- 172.21.15.107:0/18963391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca14005190 0x7fca1400b990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fca24001780 tx=0x7fca2401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bdfe04afe7e8b1b server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:27.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.735+0000 7fca2959a640 1 -- 172.21.15.107:0/18963391 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca240052d0 con 0x7fca14005190 2023-12-04T05:45:27.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.735+0000 7fca2959a640 1 -- 172.21.15.107:0/18963391 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca240a7050 con 0x7fca14005190 2023-12-04T05:45:27.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.735+0000 7fca2959a640 1 -- 172.21.15.107:0/18963391 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca24003840 con 0x7fca14005190 2023-12-04T05:45:27.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.735+0000 7fca3091a640 1 -- 172.21.15.107:0/18963391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca14005190 msgr2=0x7fca1400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:27.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.735+0000 7fca3091a640 1 --2- 172.21.15.107:0/18963391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca14005190 0x7fca1400b990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fca24001780 tx=0x7fca2401c400 comp rx=0 tx=0).stop 2023-12-04T05:45:27.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.735+0000 7fca3091a640 1 -- 172.21.15.107:0/18963391 shutdown_connections 2023-12-04T05:45:27.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.735+0000 7fca3091a640 1 --2- 172.21.15.107:0/18963391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca14005190 0x7fca1400b990 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:27.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.735+0000 7fca3091a640 1 -- 172.21.15.107:0/18963391 >> 172.21.15.107:0/18963391 conn(0x7fca2c06b6c0 msgr2=0x7fca2c06bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:27.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.735+0000 7fca3091a640 1 -- 172.21.15.107:0/18963391 shutdown_connections 2023-12-04T05:45:27.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.735+0000 7fca3091a640 1 -- 172.21.15.107:0/18963391 wait complete. 2023-12-04T05:45:27.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.735+0000 7fca3091a640 1 Processor -- start 2023-12-04T05:45:27.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.735+0000 7fca3091a640 1 -- start start 2023-12-04T05:45:27.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca3091a640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fca14005190 0x7fca2c1e3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:27.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca3091a640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fca2c157d30 0x7fca2c15a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca3091a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca2c15a7e0 0x7fca2c1452d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca3091a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c15ac60 con 0x7fca2c15a7e0 2023-12-04T05:45:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca3091a640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c15ae50 con 0x7fca14005190 2023-12-04T05:45:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca3091a640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c1e28f0 con 0x7fca2c157d30 2023-12-04T05:45:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca2ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca2c15a7e0 0x7fca2c1452d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca2ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca2c15a7e0 0x7fca2c1452d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46102/0 (socket says 172.21.15.107:46102) 2023-12-04T05:45:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca2ad9d640 1 -- 172.21.15.107:0/1407148868 learned_addr learned my addr 172.21.15.107:0/1407148868 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca2a59c640 1 --2- 172.21.15.107:0/1407148868 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fca14005190 0x7fca2c1e3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca29d9b640 1 --2- 172.21.15.107:0/1407148868 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fca2c157d30 0x7fca2c15a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca2ad9d640 1 -- 172.21.15.107:0/1407148868 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fca2c157d30 msgr2=0x7fca2c15a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca2ad9d640 1 --2- 172.21.15.107:0/1407148868 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fca2c157d30 0x7fca2c15a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca2ad9d640 1 -- 172.21.15.107:0/1407148868 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fca14005190 msgr2=0x7fca2c1e3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca2ad9d640 1 --2- 172.21.15.107:0/1407148868 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fca14005190 0x7fca2c1e3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca2ad9d640 1 -- 172.21.15.107:0/1407148868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2c1ee6a0 con 0x7fca2c15a7e0 2023-12-04T05:45:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca2ad9d640 1 --2- 172.21.15.107:0/1407148868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca2c15a7e0 0x7fca2c1452d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fca2401cca0 tx=0x7fca14005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca1affd640 1 -- 172.21.15.107:0/1407148868 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca240b0070 con 0x7fca2c15a7e0 2023-12-04T05:45:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca1affd640 1 -- 172.21.15.107:0/1407148868 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca24007a40 con 0x7fca2c15a7e0 2023-12-04T05:45:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca3091a640 1 -- 172.21.15.107:0/1407148868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca2c1ee980 con 0x7fca2c15a7e0 2023-12-04T05:45:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca1affd640 1 -- 172.21.15.107:0/1407148868 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca24004ef0 con 0x7fca2c15a7e0 2023-12-04T05:45:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca3091a640 1 -- 172.21.15.107:0/1407148868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2c1e22a0 con 0x7fca2c15a7e0 2023-12-04T05:45:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca3091a640 1 -- 172.21.15.107:0/1407148868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca2c067ec0 con 0x7fca2c15a7e0 2023-12-04T05:45:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca1affd640 1 -- 172.21.15.107:0/1407148868 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fca24007030 con 0x7fca2c15a7e0 2023-12-04T05:45:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca1affd640 1 --2- 172.21.15.107:0/1407148868 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fca0c0668d0 0x7fca0c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca2a59c640 1 --2- 172.21.15.107:0/1407148868 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fca0c0668d0 0x7fca0c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca1affd640 1 -- 172.21.15.107:0/1407148868 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6083+0+0 (secure 0 0 0) 0x7fca2411d4c0 con 0x7fca2c15a7e0 2023-12-04T05:45:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.739+0000 7fca2a59c640 1 --2- 172.21.15.107:0/1407148868 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fca0c0668d0 0x7fca0c068d90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fca1c0013e0 tx=0x7fca1c001500 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:27.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.743+0000 7fca1affd640 1 -- 172.21.15.107:0/1407148868 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fca240ea580 con 0x7fca2c15a7e0 2023-12-04T05:45:27.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.919+0000 7fca3091a640 1 -- 172.21.15.107:0/1407148868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca2c0c0170 con 0x7fca2c15a7e0 2023-12-04T05:45:27.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.919+0000 7fca1affd640 1 -- 172.21.15.107:0/1407148868 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4973 (secure 0 0 0) 0x7fca240ba090 con 0x7fca2c15a7e0 2023-12-04T05:45:27.920 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:27.920 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:25.804091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3],"up":{"mds_0":4288,"mds_1":5106,"mds_2":4263,"mds_3":4260},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:stopping","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":36,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.107:6839/3319682203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":3319682203},{"type":"v1","addr":"172.21.15.107:6839","nonce":3319682203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:27.920 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 39 2023-12-04T05:45:27.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.919+0000 7fca3091a640 1 -- 172.21.15.107:0/1407148868 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fca0c0668d0 msgr2=0x7fca0c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:27.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.919+0000 7fca3091a640 1 --2- 172.21.15.107:0/1407148868 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fca0c0668d0 0x7fca0c068d90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fca1c0013e0 tx=0x7fca1c001500 comp rx=0 tx=0).stop 2023-12-04T05:45:27.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.919+0000 7fca3091a640 1 -- 172.21.15.107:0/1407148868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca2c15a7e0 msgr2=0x7fca2c1452d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:27.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.919+0000 7fca3091a640 1 --2- 172.21.15.107:0/1407148868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca2c15a7e0 0x7fca2c1452d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fca2401cca0 tx=0x7fca14005800 comp rx=0 tx=0).stop 2023-12-04T05:45:27.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.919+0000 7fca3091a640 1 -- 172.21.15.107:0/1407148868 shutdown_connections 2023-12-04T05:45:27.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.919+0000 7fca3091a640 1 --2- 172.21.15.107:0/1407148868 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fca0c0668d0 0x7fca0c068d90 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:27.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.919+0000 7fca3091a640 1 --2- 172.21.15.107:0/1407148868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca2c15a7e0 0x7fca2c1452d0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:27.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.919+0000 7fca3091a640 1 --2- 172.21.15.107:0/1407148868 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fca2c157d30 0x7fca2c15a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:27.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.919+0000 7fca3091a640 1 --2- 172.21.15.107:0/1407148868 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fca14005190 0x7fca2c1e3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:27.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.919+0000 7fca3091a640 1 -- 172.21.15.107:0/1407148868 >> 172.21.15.107:0/1407148868 conn(0x7fca2c06b6c0 msgr2=0x7fca2c14d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:27.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.919+0000 7fca3091a640 1 -- 172.21.15.107:0/1407148868 shutdown_connections 2023-12-04T05:45:27.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:27.919+0000 7fca3091a640 1 -- 172.21.15.107:0/1407148868 wait complete. 2023-12-04T05:45:27.938 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:45:28.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~101s 2023-12-04T05:45:28.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~142s 2023-12-04T05:45:29.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:30.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.071+0000 7f7066820640 1 Processor -- start 2023-12-04T05:45:30.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.071+0000 7f7066820640 1 -- start start 2023-12-04T05:45:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.071+0000 7f7066820640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f7060154020 0x7f706015a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.071+0000 7f7066820640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70600c4c10 0x7f70600c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.071+0000 7f7066820640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706015ad50 con 0x7f70600d01c0 2023-12-04T05:45:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.071+0000 7f7066820640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70600ca230 con 0x7f7060154020 2023-12-04T05:45:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.071+0000 7f7066820640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70600c6770 con 0x7f70600c4c10 2023-12-04T05:45:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.071+0000 7f7064dbd640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70600c4c10 0x7f70600c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.071+0000 7f705ffff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f7060154020 0x7f706015a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.071+0000 7f705ffff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f7060154020 0x7f706015a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.107:46368/0 (socket says 172.21.15.107:46368) 2023-12-04T05:45:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.071+0000 7f705f7fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f70600d01c0 0x7f7060071ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:51282/0 (socket says 172.21.15.107:51282) 2023-12-04T05:45:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.071+0000 7f7064dbd640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70600c4c10 0x7f70600c5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.107:37896/0 (socket says 172.21.15.107:37896) 2023-12-04T05:45:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.071+0000 7f705ffff640 1 -- 172.21.15.107:0/1383561369 learned_addr learned my addr 172.21.15.107:0/1383561369 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:30.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.071+0000 7f7064dbd640 1 -- 172.21.15.107:0/1383561369 >> v2:172.21.15.195:3300/0 conn(0x7f7060154020 msgr2=0x7f706015a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:30.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.071+0000 7f7064dbd640 1 --2- 172.21.15.107:0/1383561369 >> v2:172.21.15.195:3300/0 conn(0x7f7060154020 0x7f706015a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:30.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.071+0000 7f7064dbd640 1 -- 172.21.15.107:0/1383561369 >> v1:172.21.15.195:6789/0 conn(0x7f70600d01c0 legacy=0x7f7060071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:30.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.071+0000 7f7064dbd640 1 -- 172.21.15.107:0/1383561369 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70600c55a0 con 0x7f70600c4c10 2023-12-04T05:45:30.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.071+0000 7f705ffff640 1 --2- 172.21.15.107:0/1383561369 >> v2:172.21.15.195:3300/0 conn(0x7f7060154020 0x7f706015a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T05:45:30.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.075+0000 7f7064dbd640 1 --2- 172.21.15.107:0/1383561369 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70600c4c10 0x7f70600c5060 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7058009d90 tx=0x7f705801f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dbcbb774c7d0942 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:30.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.075+0000 7f705effd640 1 -- 172.21.15.107:0/1383561369 <== mon.2 v2:172.21.15.195:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70580af600 con 0x7f70600c4c10 2023-12-04T05:45:30.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.075+0000 7f705effd640 1 -- 172.21.15.107:0/1383561369 <== mon.2 v2:172.21.15.195:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70580ab050 con 0x7f70600c4c10 2023-12-04T05:45:30.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.075+0000 7f705effd640 1 -- 172.21.15.107:0/1383561369 <== mon.2 v2:172.21.15.195:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70580af980 con 0x7f70600c4c10 2023-12-04T05:45:30.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.075+0000 7f7066820640 1 -- 172.21.15.107:0/1383561369 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70600c4c10 msgr2=0x7f70600c5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:30.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.075+0000 7f7066820640 1 --2- 172.21.15.107:0/1383561369 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70600c4c10 0x7f70600c5060 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7058009d90 tx=0x7f705801f130 comp rx=0 tx=0).stop 2023-12-04T05:45:30.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.075+0000 7f7066820640 1 -- 172.21.15.107:0/1383561369 shutdown_connections 2023-12-04T05:45:30.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.075+0000 7f7066820640 1 --2- 172.21.15.107:0/1383561369 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70600c4c10 0x7f70600c5060 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:30.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.075+0000 7f7066820640 1 --2- 172.21.15.107:0/1383561369 >> v2:172.21.15.195:3300/0 conn(0x7f7060154020 0x7f706015a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:30.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.075+0000 7f7066820640 1 -- 172.21.15.107:0/1383561369 >> 172.21.15.107:0/1383561369 conn(0x7f706006c6c0 msgr2=0x7f70600cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:30.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.075+0000 7f7066820640 1 -- 172.21.15.107:0/1383561369 shutdown_connections 2023-12-04T05:45:30.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.075+0000 7f7066820640 1 -- 172.21.15.107:0/1383561369 wait complete. 2023-12-04T05:45:30.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.075+0000 7f7066820640 1 Processor -- start 2023-12-04T05:45:30.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.075+0000 7f7066820640 1 -- start start 2023-12-04T05:45:30.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.075+0000 7f7066820640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70600c4c10 0x7f7060150790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:30.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.075+0000 7f7066820640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70600d01c0 0x7f706014e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:30.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.075+0000 7f7066820640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7060154020 0x7f70600ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:30.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.075+0000 7f7066820640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70601f2ad0 con 0x7f70600c4c10 2023-12-04T05:45:30.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.075+0000 7f7066820640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706015b940 con 0x7f7060154020 2023-12-04T05:45:30.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.075+0000 7f7066820640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70601f2cc0 con 0x7f70600d01c0 2023-12-04T05:45:30.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.079+0000 7f705ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70600c4c10 0x7f7060150790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:30.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.079+0000 7f705ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70600c4c10 0x7f7060150790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46118/0 (socket says 172.21.15.107:46118) 2023-12-04T05:45:30.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.079+0000 7f705ffff640 1 -- 172.21.15.107:0/3100521947 learned_addr learned my addr 172.21.15.107:0/3100521947 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:30.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.079+0000 7f7064dbd640 1 --2- 172.21.15.107:0/3100521947 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7060154020 0x7f70600ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:30.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.079+0000 7f705f7fe640 1 --2- 172.21.15.107:0/3100521947 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70600d01c0 0x7f706014e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:30.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.079+0000 7f705ffff640 1 -- 172.21.15.107:0/3100521947 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70600d01c0 msgr2=0x7f706014e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:30.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.079+0000 7f705ffff640 1 --2- 172.21.15.107:0/3100521947 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70600d01c0 0x7f706014e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:30.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.079+0000 7f705ffff640 1 -- 172.21.15.107:0/3100521947 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7060154020 msgr2=0x7f70600ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:30.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.079+0000 7f705ffff640 1 --2- 172.21.15.107:0/3100521947 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7060154020 0x7f70600ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:30.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.079+0000 7f705ffff640 1 -- 172.21.15.107:0/3100521947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70540094d0 con 0x7f70600c4c10 2023-12-04T05:45:30.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.079+0000 7f705ffff640 1 --2- 172.21.15.107:0/3100521947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70600c4c10 0x7f7060150790 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f705400e930 tx=0x7f705400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:30.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.079+0000 7f705cff9640 1 -- 172.21.15.107:0/3100521947 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705400ca00 con 0x7f70600c4c10 2023-12-04T05:45:30.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.079+0000 7f705cff9640 1 -- 172.21.15.107:0/3100521947 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705400cb60 con 0x7f70600c4c10 2023-12-04T05:45:30.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.079+0000 7f7066820640 1 -- 172.21.15.107:0/3100521947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7058009a70 con 0x7f70600c4c10 2023-12-04T05:45:30.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.079+0000 7f705cff9640 1 -- 172.21.15.107:0/3100521947 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7054013470 con 0x7f70600c4c10 2023-12-04T05:45:30.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.079+0000 7f7066820640 1 -- 172.21.15.107:0/3100521947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f706015bec0 con 0x7f70600c4c10 2023-12-04T05:45:30.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.079+0000 7f7066820640 1 -- 172.21.15.107:0/3100521947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7060067870 con 0x7f70600c4c10 2023-12-04T05:45:30.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.079+0000 7f705cff9640 1 -- 172.21.15.107:0/3100521947 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f70540135d0 con 0x7f70600c4c10 2023-12-04T05:45:30.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.079+0000 7f705cff9640 1 --2- 172.21.15.107:0/3100521947 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f704c066900 0x7f704c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:30.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.079+0000 7f705f7fe640 1 --2- 172.21.15.107:0/3100521947 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f704c066900 0x7f704c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:30.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.079+0000 7f705cff9640 1 -- 172.21.15.107:0/3100521947 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f7054084bc0 con 0x7f70600c4c10 2023-12-04T05:45:30.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.079+0000 7f705f7fe640 1 --2- 172.21.15.107:0/3100521947 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f704c066900 0x7f704c068dc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7060150d00 tx=0x7f7050007c30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:30.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.083+0000 7f705cff9640 1 -- 172.21.15.107:0/3100521947 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7054051c80 con 0x7f70600c4c10 2023-12-04T05:45:30.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.271+0000 7f7066820640 1 -- 172.21.15.107:0/3100521947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f706014e810 con 0x7f70600c4c10 2023-12-04T05:45:30.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.271+0000 7f705cff9640 1 -- 172.21.15.107:0/3100521947 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f7054017090 con 0x7f70600c4c10 2023-12-04T05:45:30.273 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:30.273 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:25.804091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3],"up":{"mds_0":4288,"mds_1":5106,"mds_2":4263,"mds_3":4260},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:stopping","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":36,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.107:6839/3319682203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":3319682203},{"type":"v1","addr":"172.21.15.107:6839","nonce":3319682203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:30.273 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 39 2023-12-04T05:45:30.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.271+0000 7f7066820640 1 -- 172.21.15.107:0/3100521947 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f704c066900 msgr2=0x7f704c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:30.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.271+0000 7f7066820640 1 --2- 172.21.15.107:0/3100521947 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f704c066900 0x7f704c068dc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7060150d00 tx=0x7f7050007c30 comp rx=0 tx=0).stop 2023-12-04T05:45:30.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.271+0000 7f7066820640 1 -- 172.21.15.107:0/3100521947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70600c4c10 msgr2=0x7f7060150790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:30.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.271+0000 7f7066820640 1 --2- 172.21.15.107:0/3100521947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70600c4c10 0x7f7060150790 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f705400e930 tx=0x7f705400edf0 comp rx=0 tx=0).stop 2023-12-04T05:45:30.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.271+0000 7f7066820640 1 -- 172.21.15.107:0/3100521947 shutdown_connections 2023-12-04T05:45:30.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.271+0000 7f7066820640 1 --2- 172.21.15.107:0/3100521947 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f704c066900 0x7f704c068dc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:30.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.271+0000 7f7066820640 1 --2- 172.21.15.107:0/3100521947 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7060154020 0x7f70600ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:30.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.271+0000 7f7066820640 1 --2- 172.21.15.107:0/3100521947 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f70600d01c0 0x7f706014e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:30.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.271+0000 7f7066820640 1 --2- 172.21.15.107:0/3100521947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70600c4c10 0x7f7060150790 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:30.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.271+0000 7f7066820640 1 -- 172.21.15.107:0/3100521947 >> 172.21.15.107:0/3100521947 conn(0x7f706006c6c0 msgr2=0x7f70600cb330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:30.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.271+0000 7f7066820640 1 -- 172.21.15.107:0/3100521947 shutdown_connections 2023-12-04T05:45:30.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:30.271+0000 7f7066820640 1 -- 172.21.15.107:0/3100521947 wait complete. 2023-12-04T05:45:30.293 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-12-04T05:45:30.294 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 39, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-12-04T05:39:59.032019+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 39, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3 ], "info": { "gid_4260": { "addr": "172.21.15.107:6836/3444851515", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 3444851515, "type": "v2" }, { "addr": "172.21.15.107:6836", "nonce": 3444851515, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4260, "incarnation": 14, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:stopping", "state_seq": 9 }, "gid_4263": { "addr": "172.21.15.107:6837/2210527429", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6835", "nonce": 2210527429, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 2210527429, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4263, "incarnation": 12, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 7 }, "gid_4288": { "addr": "172.21.15.195:6837/3412865477", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6835", "nonce": 3412865477, "type": "v2" }, { "addr": "172.21.15.195:6837", "nonce": 3412865477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_5106": { "addr": "172.21.15.107:6839/3319682203", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6838", "nonce": 3319682203, "type": "v2" }, { "addr": "172.21.15.107:6839", "nonce": 3319682203, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5106, "incarnation": 36, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:rejoin", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 26, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-12-04T05:45:25.804091+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 4 ], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 5106, "mds_2": 4263, "mds_3": 4260 } } } ], "standbys": [] } 2023-12-04T05:45:30.697 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:45:30.699 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:45:32.295 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:32.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.415+0000 7f202b49e640 1 Processor -- start 2023-12-04T05:45:32.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.419+0000 7f202b49e640 1 -- start start 2023-12-04T05:45:32.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.419+0000 7f202b49e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2024072110 0x7f2024158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:32.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.419+0000 7f202b49e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f20240724f0 0x7f20240c7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.419+0000 7f202b49e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f20240c56f0 0x7f20240c7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.419+0000 7f202b49e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20240c7ea0 con 0x7f2024072110 2023-12-04T05:45:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.419+0000 7f202b49e640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2024158da0 con 0x7f20240c56f0 2023-12-04T05:45:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.419+0000 7f202b49e640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20241583d0 con 0x7f20240724f0 2023-12-04T05:45:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.419+0000 7f202923a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2024072110 0x7f2024158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.419+0000 7f202923a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2024072110 0x7f2024158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46122/0 (socket says 172.21.15.107:46122) 2023-12-04T05:45:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.419+0000 7f202923a640 1 -- 172.21.15.107:0/30977009 learned_addr learned my addr 172.21.15.107:0/30977009 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.419+0000 7f2029a3b640 1 --2- 172.21.15.107:0/30977009 >> v2:172.21.15.195:3300/0 conn(0x7f20240c56f0 0x7f20240c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.419+0000 7f2028a39640 1 --2- 172.21.15.107:0/30977009 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f20240724f0 0x7f20240c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.419+0000 7f202923a640 1 -- 172.21.15.107:0/30977009 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f20240724f0 msgr2=0x7f20240c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.419+0000 7f202923a640 1 --2- 172.21.15.107:0/30977009 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f20240724f0 0x7f20240c7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.419+0000 7f202923a640 1 -- 172.21.15.107:0/30977009 >> v2:172.21.15.195:3300/0 conn(0x7f20240c56f0 msgr2=0x7f20240c7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.419+0000 7f202923a640 1 --2- 172.21.15.107:0/30977009 >> v2:172.21.15.195:3300/0 conn(0x7f20240c56f0 0x7f20240c7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.419+0000 7f202923a640 1 -- 172.21.15.107:0/30977009 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20240c6170 con 0x7f2024072110 2023-12-04T05:45:32.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.419+0000 7f202923a640 1 --2- 172.21.15.107:0/30977009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2024072110 0x7f2024158860 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f202406f4a0 tx=0x7f202001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7902d06e6b7a8fd0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:32.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.419+0000 7f201bfff640 1 -- 172.21.15.107:0/30977009 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20200a89b0 con 0x7f2024072110 2023-12-04T05:45:32.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.419+0000 7f201bfff640 1 -- 172.21.15.107:0/30977009 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20200a8b10 con 0x7f2024072110 2023-12-04T05:45:32.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.419+0000 7f201bfff640 1 -- 172.21.15.107:0/30977009 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20200a8ea0 con 0x7f2024072110 2023-12-04T05:45:32.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202b49e640 1 -- 172.21.15.107:0/30977009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2024072110 msgr2=0x7f2024158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:32.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202b49e640 1 --2- 172.21.15.107:0/30977009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2024072110 0x7f2024158860 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f202406f4a0 tx=0x7f202001ecf0 comp rx=0 tx=0).stop 2023-12-04T05:45:32.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202b49e640 1 -- 172.21.15.107:0/30977009 shutdown_connections 2023-12-04T05:45:32.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202b49e640 1 --2- 172.21.15.107:0/30977009 >> v2:172.21.15.195:3300/0 conn(0x7f20240c56f0 0x7f20240c7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:32.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202b49e640 1 --2- 172.21.15.107:0/30977009 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f20240724f0 0x7f20240c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:32.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202b49e640 1 --2- 172.21.15.107:0/30977009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2024072110 0x7f2024158860 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:32.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202b49e640 1 -- 172.21.15.107:0/30977009 >> 172.21.15.107:0/30977009 conn(0x7f20240c7ff0 msgr2=0x7f20240c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:32.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202b49e640 1 -- 172.21.15.107:0/30977009 shutdown_connections 2023-12-04T05:45:32.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202b49e640 1 -- 172.21.15.107:0/30977009 wait complete. 2023-12-04T05:45:32.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202b49e640 1 Processor -- start 2023-12-04T05:45:32.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202b49e640 1 -- start start 2023-12-04T05:45:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202b49e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2024072110 0x7f202414cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202b49e640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f20240724f0 0x7f202406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202b49e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f20240c56f0 0x7f202415b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202b49e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2024154d60 con 0x7f2024072110 2023-12-04T05:45:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202b49e640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20241eee90 con 0x7f20240724f0 2023-12-04T05:45:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202b49e640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2024159450 con 0x7f20240c56f0 2023-12-04T05:45:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202923a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2024072110 0x7f202414cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202923a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2024072110 0x7f202414cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46124/0 (socket says 172.21.15.107:46124) 2023-12-04T05:45:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202923a640 1 -- 172.21.15.107:0/617635490 learned_addr learned my addr 172.21.15.107:0/617635490 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f2029a3b640 1 --2- 172.21.15.107:0/617635490 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f20240c56f0 0x7f202415b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f2028a39640 1 --2- 172.21.15.107:0/617635490 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f20240724f0 0x7f202406d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202923a640 1 -- 172.21.15.107:0/617635490 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f20240c56f0 msgr2=0x7f202415b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202923a640 1 --2- 172.21.15.107:0/617635490 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f20240c56f0 0x7f202415b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202923a640 1 -- 172.21.15.107:0/617635490 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f20240724f0 msgr2=0x7f202406d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202923a640 1 --2- 172.21.15.107:0/617635490 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f20240724f0 0x7f202406d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202923a640 1 -- 172.21.15.107:0/617635490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20200094d0 con 0x7f2024072110 2023-12-04T05:45:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f2028a39640 1 --2- 172.21.15.107:0/617635490 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f20240724f0 0x7f202406d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:45:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202923a640 1 --2- 172.21.15.107:0/617635490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2024072110 0x7f202414cb40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f202001d3a0 tx=0x7f202001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f2019ffb640 1 -- 172.21.15.107:0/617635490 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20200ba600 con 0x7f2024072110 2023-12-04T05:45:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f2019ffb640 1 -- 172.21.15.107:0/617635490 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20200ba760 con 0x7f2024072110 2023-12-04T05:45:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202b49e640 1 -- 172.21.15.107:0/617635490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f202414c830 con 0x7f2024072110 2023-12-04T05:45:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f2019ffb640 1 -- 172.21.15.107:0/617635490 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20200bab10 con 0x7f2024072110 2023-12-04T05:45:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.423+0000 7f202b49e640 1 -- 172.21.15.107:0/617635490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20241f1780 con 0x7f2024072110 2023-12-04T05:45:32.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.427+0000 7f202b49e640 1 -- 172.21.15.107:0/617635490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2024067870 con 0x7f2024072110 2023-12-04T05:45:32.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.427+0000 7f2019ffb640 1 -- 172.21.15.107:0/617635490 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f20200bac70 con 0x7f2024072110 2023-12-04T05:45:32.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.427+0000 7f2019ffb640 1 --2- 172.21.15.107:0/617635490 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2008066900 0x7f2008068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:32.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.427+0000 7f2028a39640 1 --2- 172.21.15.107:0/617635490 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2008066900 0x7f2008068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:32.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.427+0000 7f2019ffb640 1 -- 172.21.15.107:0/617635490 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f2020122af0 con 0x7f2024072110 2023-12-04T05:45:32.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.427+0000 7f2028a39640 1 --2- 172.21.15.107:0/617635490 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2008066900 0x7f2008068dc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f20241e7cb0 tx=0x7f2014008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:32.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.431+0000 7f2019ffb640 1 -- 172.21.15.107:0/617635490 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f20200efd60 con 0x7f2024072110 2023-12-04T05:45:32.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.607+0000 7f202b49e640 1 -- 172.21.15.107:0/617635490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20241519c0 con 0x7f2024072110 2023-12-04T05:45:32.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.607+0000 7f2019ffb640 1 -- 172.21.15.107:0/617635490 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f20200b5090 con 0x7f2024072110 2023-12-04T05:45:32.610 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:32.610 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:30.866978+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3],"up":{"mds_0":4288,"mds_1":5106,"mds_2":4263,"mds_3":4260},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:stopping","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":36,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6839/3319682203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":3319682203},{"type":"v1","addr":"172.21.15.107:6839","nonce":3319682203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:32.610 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 40 2023-12-04T05:45:32.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.611+0000 7f202b49e640 1 -- 172.21.15.107:0/617635490 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2008066900 msgr2=0x7f2008068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:32.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.611+0000 7f202b49e640 1 --2- 172.21.15.107:0/617635490 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2008066900 0x7f2008068dc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f20241e7cb0 tx=0x7f2014008040 comp rx=0 tx=0).stop 2023-12-04T05:45:32.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.611+0000 7f202b49e640 1 -- 172.21.15.107:0/617635490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2024072110 msgr2=0x7f202414cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:32.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.611+0000 7f202b49e640 1 --2- 172.21.15.107:0/617635490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2024072110 0x7f202414cb40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f202001d3a0 tx=0x7f202001ba90 comp rx=0 tx=0).stop 2023-12-04T05:45:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.611+0000 7f202b49e640 1 -- 172.21.15.107:0/617635490 shutdown_connections 2023-12-04T05:45:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.611+0000 7f202b49e640 1 --2- 172.21.15.107:0/617635490 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2008066900 0x7f2008068dc0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.611+0000 7f202b49e640 1 --2- 172.21.15.107:0/617635490 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f20240c56f0 0x7f202415b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.611+0000 7f202b49e640 1 --2- 172.21.15.107:0/617635490 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f20240724f0 0x7f202406d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.611+0000 7f202b49e640 1 --2- 172.21.15.107:0/617635490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2024072110 0x7f202414cb40 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.611+0000 7f202b49e640 1 -- 172.21.15.107:0/617635490 >> 172.21.15.107:0/617635490 conn(0x7f20240c7ff0 msgr2=0x7f20240c7630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.611+0000 7f202b49e640 1 -- 172.21.15.107:0/617635490 shutdown_connections 2023-12-04T05:45:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:32.611+0000 7f202b49e640 1 -- 172.21.15.107:0/617635490 wait complete. 2023-12-04T05:45:32.626 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:45:34.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:34.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.755+0000 7f0a685b0640 1 Processor -- start 2023-12-04T05:45:34.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.755+0000 7f0a685b0640 1 -- start start 2023-12-04T05:45:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.759+0000 7f0a685b0640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0a60072110 0x7f0a60158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.759+0000 7f0a685b0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0a600c56f0 0x7f0a60140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.759+0000 7f0a685b0640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a60158da0 con 0x7f0a600724f0 2023-12-04T05:45:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.759+0000 7f0a685b0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a601583d0 con 0x7f0a600c56f0 2023-12-04T05:45:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.759+0000 7f0a685b0640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a600c6170 con 0x7f0a60072110 2023-12-04T05:45:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.759+0000 7f0a66b4d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0a600c56f0 0x7f0a60140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.759+0000 7f0a66b4d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0a600c56f0 0x7f0a60140830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46138/0 (socket says 172.21.15.107:46138) 2023-12-04T05:45:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.759+0000 7f0a66b4d640 1 -- 172.21.15.107:0/1786822620 learned_addr learned my addr 172.21.15.107:0/1786822620 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.759+0000 7f0a6634c640 1 --2- 172.21.15.107:0/1786822620 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0a60072110 0x7f0a60158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.759+0000 7f0a66b4d640 1 -- 172.21.15.107:0/1786822620 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0a60072110 msgr2=0x7f0a60158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:34.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.759+0000 7f0a66b4d640 1 --2- 172.21.15.107:0/1786822620 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0a60072110 0x7f0a60158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:34.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.759+0000 7f0a66b4d640 1 -- 172.21.15.107:0/1786822620 >> v1:172.21.15.195:6789/0 conn(0x7f0a600724f0 legacy=0x7f0a600c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:34.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.759+0000 7f0a66b4d640 1 -- 172.21.15.107:0/1786822620 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a60157da0 con 0x7f0a600c56f0 2023-12-04T05:45:34.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.759+0000 7f0a66b4d640 1 --2- 172.21.15.107:0/1786822620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a600c56f0 0x7f0a60140830 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0a58009e50 tx=0x7f0a5801c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a178d00e7f664283 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:34.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.759+0000 7f0a6534a640 1 -- 172.21.15.107:0/1786822620 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a580a5740 con 0x7f0a600c56f0 2023-12-04T05:45:34.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.759+0000 7f0a6534a640 1 -- 172.21.15.107:0/1786822620 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a580a58a0 con 0x7f0a600c56f0 2023-12-04T05:45:34.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.759+0000 7f0a6534a640 1 -- 172.21.15.107:0/1786822620 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a580a5c30 con 0x7f0a600c56f0 2023-12-04T05:45:34.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a685b0640 1 -- 172.21.15.107:0/1786822620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a600c56f0 msgr2=0x7f0a60140830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:34.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a685b0640 1 --2- 172.21.15.107:0/1786822620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a600c56f0 0x7f0a60140830 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0a58009e50 tx=0x7f0a5801c9d0 comp rx=0 tx=0).stop 2023-12-04T05:45:34.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a685b0640 1 -- 172.21.15.107:0/1786822620 shutdown_connections 2023-12-04T05:45:34.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a685b0640 1 --2- 172.21.15.107:0/1786822620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a600c56f0 0x7f0a60140830 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:34.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a685b0640 1 --2- 172.21.15.107:0/1786822620 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0a60072110 0x7f0a60158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:34.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a685b0640 1 -- 172.21.15.107:0/1786822620 >> 172.21.15.107:0/1786822620 conn(0x7f0a600c7ff0 msgr2=0x7f0a600c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:34.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a685b0640 1 -- 172.21.15.107:0/1786822620 shutdown_connections 2023-12-04T05:45:34.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a685b0640 1 -- 172.21.15.107:0/1786822620 wait complete. 2023-12-04T05:45:34.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a685b0640 1 Processor -- start 2023-12-04T05:45:34.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a685b0640 1 -- start start 2023-12-04T05:45:34.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a685b0640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a60072110 0x7f0a601367a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:34.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a685b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a600724f0 0x7f0a6013dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:34.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a685b0640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0a600c56f0 0x7f0a6013b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:34.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a685b0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a60131b20 con 0x7f0a600724f0 2023-12-04T05:45:34.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a685b0640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6013e620 con 0x7f0a60072110 2023-12-04T05:45:34.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a685b0640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6013e900 con 0x7f0a600c56f0 2023-12-04T05:45:34.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a65b4b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a600724f0 0x7f0a6013dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:34.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a65b4b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a600724f0 0x7f0a6013dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46150/0 (socket says 172.21.15.107:46150) 2023-12-04T05:45:34.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a65b4b640 1 -- 172.21.15.107:0/2531485687 learned_addr learned my addr 172.21.15.107:0/2531485687 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:34.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a66b4d640 1 --2- 172.21.15.107:0/2531485687 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0a600c56f0 0x7f0a6013b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:34.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a6634c640 1 --2- 172.21.15.107:0/2531485687 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a60072110 0x7f0a601367a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:34.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a65b4b640 1 -- 172.21.15.107:0/2531485687 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0a600c56f0 msgr2=0x7f0a6013b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:34.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a65b4b640 1 --2- 172.21.15.107:0/2531485687 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0a600c56f0 0x7f0a6013b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:34.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a65b4b640 1 -- 172.21.15.107:0/2531485687 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a60072110 msgr2=0x7f0a601367a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:34.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a65b4b640 1 --2- 172.21.15.107:0/2531485687 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a60072110 0x7f0a601367a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:34.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a65b4b640 1 -- 172.21.15.107:0/2531485687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a58009b30 con 0x7f0a600724f0 2023-12-04T05:45:34.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a65b4b640 1 --2- 172.21.15.107:0/2531485687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a600724f0 0x7f0a6013dc70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c00c410 tx=0x7f0a5c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:34.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a56ffd640 1 -- 172.21.15.107:0/2531485687 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c011790 con 0x7f0a600724f0 2023-12-04T05:45:34.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a56ffd640 1 -- 172.21.15.107:0/2531485687 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5c0118f0 con 0x7f0a600724f0 2023-12-04T05:45:34.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a685b0640 1 -- 172.21.15.107:0/2531485687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6013ebe0 con 0x7f0a600724f0 2023-12-04T05:45:34.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a56ffd640 1 -- 172.21.15.107:0/2531485687 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c011ce0 con 0x7f0a600724f0 2023-12-04T05:45:34.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.763+0000 7f0a685b0640 1 -- 172.21.15.107:0/2531485687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a601409b0 con 0x7f0a600724f0 2023-12-04T05:45:34.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.767+0000 7f0a685b0640 1 -- 172.21.15.107:0/2531485687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a60067870 con 0x7f0a600724f0 2023-12-04T05:45:34.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.767+0000 7f0a56ffd640 1 -- 172.21.15.107:0/2531485687 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f0a5c01c460 con 0x7f0a600724f0 2023-12-04T05:45:34.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.767+0000 7f0a56ffd640 1 --2- 172.21.15.107:0/2531485687 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0a440668d0 0x7f0a44068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:34.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.767+0000 7f0a56ffd640 1 -- 172.21.15.107:0/2531485687 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f0a5c084270 con 0x7f0a600724f0 2023-12-04T05:45:34.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.767+0000 7f0a6634c640 1 --2- 172.21.15.107:0/2531485687 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0a440668d0 0x7f0a44068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:34.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.767+0000 7f0a6634c640 1 --2- 172.21.15.107:0/2531485687 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0a440668d0 0x7f0a44068d90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0a5000b9d0 tx=0x7f0a50008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:34.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.771+0000 7f0a56ffd640 1 -- 172.21.15.107:0/2531485687 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0a5c0513c0 con 0x7f0a600724f0 2023-12-04T05:45:34.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.963+0000 7f0a685b0640 1 -- 172.21.15.107:0/2531485687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a600c7400 con 0x7f0a600724f0 2023-12-04T05:45:34.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.963+0000 7f0a56ffd640 1 -- 172.21.15.107:0/2531485687 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f0a5c0549e0 con 0x7f0a600724f0 2023-12-04T05:45:34.965 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:34.965 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:30.866978+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3],"up":{"mds_0":4288,"mds_1":5106,"mds_2":4263,"mds_3":4260},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:stopping","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":36,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6839/3319682203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":3319682203},{"type":"v1","addr":"172.21.15.107:6839","nonce":3319682203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:34.965 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 40 2023-12-04T05:45:34.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.963+0000 7f0a685b0640 1 -- 172.21.15.107:0/2531485687 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0a440668d0 msgr2=0x7f0a44068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:34.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.963+0000 7f0a685b0640 1 --2- 172.21.15.107:0/2531485687 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0a440668d0 0x7f0a44068d90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0a5000b9d0 tx=0x7f0a50008040 comp rx=0 tx=0).stop 2023-12-04T05:45:34.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.963+0000 7f0a685b0640 1 -- 172.21.15.107:0/2531485687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a600724f0 msgr2=0x7f0a6013dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:34.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.963+0000 7f0a685b0640 1 --2- 172.21.15.107:0/2531485687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a600724f0 0x7f0a6013dc70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c00c410 tx=0x7f0a5c00c8d0 comp rx=0 tx=0).stop 2023-12-04T05:45:34.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.963+0000 7f0a685b0640 1 -- 172.21.15.107:0/2531485687 shutdown_connections 2023-12-04T05:45:34.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.963+0000 7f0a685b0640 1 --2- 172.21.15.107:0/2531485687 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0a440668d0 0x7f0a44068d90 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:34.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.963+0000 7f0a685b0640 1 --2- 172.21.15.107:0/2531485687 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0a600c56f0 0x7f0a6013b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:34.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.963+0000 7f0a685b0640 1 --2- 172.21.15.107:0/2531485687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a600724f0 0x7f0a6013dc70 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:34.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.963+0000 7f0a685b0640 1 --2- 172.21.15.107:0/2531485687 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a60072110 0x7f0a601367a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:34.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.963+0000 7f0a685b0640 1 -- 172.21.15.107:0/2531485687 >> 172.21.15.107:0/2531485687 conn(0x7f0a600c7ff0 msgr2=0x7f0a60157580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:34.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.967+0000 7f0a685b0640 1 -- 172.21.15.107:0/2531485687 shutdown_connections 2023-12-04T05:45:34.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:34.967+0000 7f0a685b0640 1 -- 172.21.15.107:0/2531485687 wait complete. 2023-12-04T05:45:34.984 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:45:35.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~107s 2023-12-04T05:45:35.132 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~148s 2023-12-04T05:45:35.285 INFO:tasks.ceph.mds.f.smithi195.stderr:2023-12-04T05:45:35.282+0000 7f5bdadb8640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-12-04T05:45:36.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:37.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.119+0000 7fc00a5c2640 1 Processor -- start 2023-12-04T05:45:37.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.119+0000 7fc00a5c2640 1 -- start start 2023-12-04T05:45:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.119+0000 7fc00a5c2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc0040c46a0 0x7fc0040c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.119+0000 7fc00a5c2640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc00406e660 con 0x7fc004157180 2023-12-04T05:45:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.119+0000 7fc00a5c2640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc004156cf0 con 0x7fc0040c4a80 2023-12-04T05:45:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.119+0000 7fc00a5c2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc004156490 con 0x7fc0040c46a0 2023-12-04T05:45:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.119+0000 7fc003fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc0040c46a0 0x7fc0040c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.119+0000 7fc003fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc0040c46a0 0x7fc0040c8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46160/0 (socket says 172.21.15.107:46160) 2023-12-04T05:45:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.119+0000 7fc003fff640 1 -- 172.21.15.107:0/1197764141 learned_addr learned my addr 172.21.15.107:0/1197764141 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.119+0000 7fbffb7fe640 1 --1- 172.21.15.107:0/1197764141 >> v1:172.21.15.107:6789/0 conn(0x7fc004157180 0x7fc0040c5ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59686/0 (socket says 172.21.15.107:59686) 2023-12-04T05:45:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.119+0000 7fc003fff640 1 -- 172.21.15.107:0/1197764141 >> v1:172.21.15.195:6789/0 conn(0x7fc0040c4a80 legacy=0x7fc0040c50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.119+0000 7fc003fff640 1 -- 172.21.15.107:0/1197764141 >> v1:172.21.15.107:6789/0 conn(0x7fc004157180 legacy=0x7fc0040c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.119+0000 7fc003fff640 1 -- 172.21.15.107:0/1197764141 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00406e0a0 con 0x7fc0040c46a0 2023-12-04T05:45:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.119+0000 7fc0037fe640 1 -- 172.21.15.107:0/1197764141 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522801311 0 0) 0x7fbfec001120 con 0x7fc004157180 2023-12-04T05:45:37.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc003fff640 1 --2- 172.21.15.107:0/1197764141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0040c46a0 0x7fc0040c8b10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbff4009870 tx=0x7fbff401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a1738efed29a5 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:37.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc0037fe640 1 -- 172.21.15.107:0/1197764141 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff40ab5f0 con 0x7fc0040c46a0 2023-12-04T05:45:37.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc0037fe640 1 -- 172.21.15.107:0/1197764141 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff40ab750 con 0x7fc0040c46a0 2023-12-04T05:45:37.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc0037fe640 1 -- 172.21.15.107:0/1197764141 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff40abae0 con 0x7fc0040c46a0 2023-12-04T05:45:37.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc00a5c2640 1 -- 172.21.15.107:0/1197764141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0040c46a0 msgr2=0x7fc0040c8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:37.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc00a5c2640 1 --2- 172.21.15.107:0/1197764141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0040c46a0 0x7fc0040c8b10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbff4009870 tx=0x7fbff401c500 comp rx=0 tx=0).stop 2023-12-04T05:45:37.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc00a5c2640 1 -- 172.21.15.107:0/1197764141 shutdown_connections 2023-12-04T05:45:37.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc00a5c2640 1 --2- 172.21.15.107:0/1197764141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0040c46a0 0x7fc0040c8b10 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:37.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc00a5c2640 1 -- 172.21.15.107:0/1197764141 >> 172.21.15.107:0/1197764141 conn(0x7fc0040c7ff0 msgr2=0x7fc0040c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:37.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc00a5c2640 1 -- 172.21.15.107:0/1197764141 shutdown_connections 2023-12-04T05:45:37.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc00a5c2640 1 -- 172.21.15.107:0/1197764141 wait complete. 2023-12-04T05:45:37.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc00a5c2640 1 Processor -- start 2023-12-04T05:45:37.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc00a5c2640 1 -- start start 2023-12-04T05:45:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc00a5c2640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0040c46a0 0x7fc00414b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc00a5c2640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc0040c4a80 0x7fc0041527d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc00a5c2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc004157180 0x7fc004151b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc00a5c2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc004152280 con 0x7fc004157180 2023-12-04T05:45:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc00a5c2640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc004151660 con 0x7fc0040c46a0 2023-12-04T05:45:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc00a5c2640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0041538d0 con 0x7fc0040c4a80 2023-12-04T05:45:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc008b5f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc004157180 0x7fc004151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc008b5f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc004157180 0x7fc004151b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46176/0 (socket says 172.21.15.107:46176) 2023-12-04T05:45:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc008b5f640 1 -- 172.21.15.107:0/253207629 learned_addr learned my addr 172.21.15.107:0/253207629 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc003fff640 1 --2- 172.21.15.107:0/253207629 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0040c46a0 0x7fc00414b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fbffb7fe640 1 --2- 172.21.15.107:0/253207629 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc0040c4a80 0x7fc0041527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc008b5f640 1 -- 172.21.15.107:0/253207629 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc0040c4a80 msgr2=0x7fc0041527d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc008b5f640 1 --2- 172.21.15.107:0/253207629 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc0040c4a80 0x7fc0041527d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc008b5f640 1 -- 172.21.15.107:0/253207629 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0040c46a0 msgr2=0x7fc00414b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc008b5f640 1 --2- 172.21.15.107:0/253207629 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0040c46a0 0x7fc00414b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc008b5f640 1 -- 172.21.15.107:0/253207629 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff40094d0 con 0x7fc004157180 2023-12-04T05:45:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.123+0000 7fc008b5f640 1 --2- 172.21.15.107:0/253207629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc004157180 0x7fc004151b90 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fbffc00b970 tx=0x7fbffc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.127+0000 7fc0017fa640 1 -- 172.21.15.107:0/253207629 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc00cb40 con 0x7fc004157180 2023-12-04T05:45:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.127+0000 7fc0017fa640 1 -- 172.21.15.107:0/253207629 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbffc00cca0 con 0x7fc004157180 2023-12-04T05:45:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.127+0000 7fc00a5c2640 1 -- 172.21.15.107:0/253207629 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0041f6bc0 con 0x7fc004157180 2023-12-04T05:45:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.127+0000 7fc0017fa640 1 -- 172.21.15.107:0/253207629 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc014680 con 0x7fc004157180 2023-12-04T05:45:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.127+0000 7fc00a5c2640 1 -- 172.21.15.107:0/253207629 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0041518b0 con 0x7fc004157180 2023-12-04T05:45:37.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.127+0000 7fc00a5c2640 1 -- 172.21.15.107:0/253207629 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc004067870 con 0x7fc004157180 2023-12-04T05:45:37.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.127+0000 7fc0017fa640 1 -- 172.21.15.107:0/253207629 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fbffc01d3b0 con 0x7fc004157180 2023-12-04T05:45:37.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.127+0000 7fc0017fa640 1 --2- 172.21.15.107:0/253207629 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbfe40668d0 0x7fbfe4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:37.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.127+0000 7fc0017fa640 1 -- 172.21.15.107:0/253207629 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6083+0+0 (secure 0 0 0) 0x7fbffc084470 con 0x7fc004157180 2023-12-04T05:45:37.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.127+0000 7fc003fff640 1 --2- 172.21.15.107:0/253207629 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbfe40668d0 0x7fbfe4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:37.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.127+0000 7fc003fff640 1 --2- 172.21.15.107:0/253207629 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbfe40668d0 0x7fbfe4068d90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbff40af000 tx=0x7fbff40a7000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:37.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.131+0000 7fc0017fa640 1 -- 172.21.15.107:0/253207629 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbffc051490 con 0x7fc004157180 2023-12-04T05:45:37.302 INFO:tasks.ceph.mds.a.smithi107.stderr:./src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7fd9ace98640 time 2023-12-04T05:45:37.302432+0000 2023-12-04T05:45:37.302 INFO:tasks.ceph.mds.a.smithi107.stderr:./src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-12-04T05:45:37.303 INFO:tasks.ceph.mds.a.smithi107.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:45:37.303 INFO:tasks.ceph.mds.a.smithi107.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7fd9b2fc5e96] 2023-12-04T05:45:37.303 INFO:tasks.ceph.mds.a.smithi107.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7fd9b2fc604a] 2023-12-04T05:45:37.303 INFO:tasks.ceph.mds.a.smithi107.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x5636330593e8] 2023-12-04T05:45:37.303 INFO:tasks.ceph.mds.a.smithi107.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x56363304e174] 2023-12-04T05:45:37.303 INFO:tasks.ceph.mds.a.smithi107.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x563632fff033] 2023-12-04T05:45:37.303 INFO:tasks.ceph.mds.a.smithi107.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x563632f39dec] 2023-12-04T05:45:37.303 INFO:tasks.ceph.mds.a.smithi107.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x563632f40be8] 2023-12-04T05:45:37.303 INFO:tasks.ceph.mds.a.smithi107.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x563632f36a91] 2023-12-04T05:45:37.303 INFO:tasks.ceph.mds.a.smithi107.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x563632f3734c] 2023-12-04T05:45:37.304 INFO:tasks.ceph.mds.a.smithi107.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563632f23837] 2023-12-04T05:45:37.304 INFO:tasks.ceph.mds.a.smithi107.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x563632ea75fb] 2023-12-04T05:45:37.304 INFO:tasks.ceph.mds.a.smithi107.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563632ea7acc] 2023-12-04T05:45:37.304 INFO:tasks.ceph.mds.a.smithi107.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x563632e928e5] 2023-12-04T05:45:37.304 INFO:tasks.ceph.mds.a.smithi107.stderr: 14: (DispatchQueue::entry()+0x5ea) [0x7fd9b31b211a] 2023-12-04T05:45:37.304 INFO:tasks.ceph.mds.a.smithi107.stderr: 15: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7fd9b3244991] 2023-12-04T05:45:37.304 INFO:tasks.ceph.mds.a.smithi107.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fd9b2952b43] 2023-12-04T05:45:37.304 INFO:tasks.ceph.mds.a.smithi107.stderr: 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fd9b29e4a00] 2023-12-04T05:45:37.304 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-12-04T05:45:37.303+0000 7fd9ace98640 -1 ./src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7fd9ace98640 time 2023-12-04T05:45:37.302432+0000 2023-12-04T05:45:37.304 INFO:tasks.ceph.mds.a.smithi107.stderr:./src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-12-04T05:45:37.304 INFO:tasks.ceph.mds.a.smithi107.stderr: 2023-12-04T05:45:37.304 INFO:tasks.ceph.mds.a.smithi107.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:45:37.304 INFO:tasks.ceph.mds.a.smithi107.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7fd9b2fc5e96] 2023-12-04T05:45:37.304 INFO:tasks.ceph.mds.a.smithi107.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7fd9b2fc604a] 2023-12-04T05:45:37.304 INFO:tasks.ceph.mds.a.smithi107.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x5636330593e8] 2023-12-04T05:45:37.304 INFO:tasks.ceph.mds.a.smithi107.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x56363304e174] 2023-12-04T05:45:37.305 INFO:tasks.ceph.mds.a.smithi107.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x563632fff033] 2023-12-04T05:45:37.305 INFO:tasks.ceph.mds.a.smithi107.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x563632f39dec] 2023-12-04T05:45:37.305 INFO:tasks.ceph.mds.a.smithi107.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x563632f40be8] 2023-12-04T05:45:37.305 INFO:tasks.ceph.mds.a.smithi107.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x563632f36a91] 2023-12-04T05:45:37.305 INFO:tasks.ceph.mds.a.smithi107.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x563632f3734c] 2023-12-04T05:45:37.305 INFO:tasks.ceph.mds.a.smithi107.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563632f23837] 2023-12-04T05:45:37.305 INFO:tasks.ceph.mds.a.smithi107.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x563632ea75fb] 2023-12-04T05:45:37.305 INFO:tasks.ceph.mds.a.smithi107.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563632ea7acc] 2023-12-04T05:45:37.305 INFO:tasks.ceph.mds.a.smithi107.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x563632e928e5] 2023-12-04T05:45:37.305 INFO:tasks.ceph.mds.a.smithi107.stderr: 14: (DispatchQueue::entry()+0x5ea) [0x7fd9b31b211a] 2023-12-04T05:45:37.305 INFO:tasks.ceph.mds.a.smithi107.stderr: 15: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7fd9b3244991] 2023-12-04T05:45:37.305 INFO:tasks.ceph.mds.a.smithi107.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fd9b2952b43] 2023-12-04T05:45:37.305 INFO:tasks.ceph.mds.a.smithi107.stderr: 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fd9b29e4a00] 2023-12-04T05:45:37.305 INFO:tasks.ceph.mds.a.smithi107.stderr: 2023-12-04T05:45:37.305 INFO:tasks.ceph.mds.a.smithi107.stderr:*** Caught signal (Aborted) ** 2023-12-04T05:45:37.306 INFO:tasks.ceph.mds.a.smithi107.stderr: in thread 7fd9ace98640 thread_name:ms_dispatch 2023-12-04T05:45:37.306 INFO:tasks.ceph.mds.a.smithi107.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:45:37.306 INFO:tasks.ceph.mds.a.smithi107.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fd9b2900520] 2023-12-04T05:45:37.306 INFO:tasks.ceph.mds.a.smithi107.stderr: 2: pthread_kill() 2023-12-04T05:45:37.306 INFO:tasks.ceph.mds.a.smithi107.stderr: 3: raise() 2023-12-04T05:45:37.306 INFO:tasks.ceph.mds.a.smithi107.stderr: 4: abort() 2023-12-04T05:45:37.306 INFO:tasks.ceph.mds.a.smithi107.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fd9b2fc5ef0] 2023-12-04T05:45:37.306 INFO:tasks.ceph.mds.a.smithi107.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7fd9b2fc604a] 2023-12-04T05:45:37.307 INFO:tasks.ceph.mds.a.smithi107.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x5636330593e8] 2023-12-04T05:45:37.307 INFO:tasks.ceph.mds.a.smithi107.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x56363304e174] 2023-12-04T05:45:37.307 INFO:tasks.ceph.mds.a.smithi107.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x563632fff033] 2023-12-04T05:45:37.307 INFO:tasks.ceph.mds.a.smithi107.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x563632f39dec] 2023-12-04T05:45:37.307 INFO:tasks.ceph.mds.a.smithi107.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x563632f40be8] 2023-12-04T05:45:37.307 INFO:tasks.ceph.mds.a.smithi107.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x563632f36a91] 2023-12-04T05:45:37.307 INFO:tasks.ceph.mds.a.smithi107.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x563632f3734c] 2023-12-04T05:45:37.307 INFO:tasks.ceph.mds.a.smithi107.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563632f23837] 2023-12-04T05:45:37.307 INFO:tasks.ceph.mds.a.smithi107.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x563632ea75fb] 2023-12-04T05:45:37.307 INFO:tasks.ceph.mds.a.smithi107.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563632ea7acc] 2023-12-04T05:45:37.307 INFO:tasks.ceph.mds.a.smithi107.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x563632e928e5] 2023-12-04T05:45:37.307 INFO:tasks.ceph.mds.a.smithi107.stderr: 18: (DispatchQueue::entry()+0x5ea) [0x7fd9b31b211a] 2023-12-04T05:45:37.307 INFO:tasks.ceph.mds.a.smithi107.stderr: 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7fd9b3244991] 2023-12-04T05:45:37.307 INFO:tasks.ceph.mds.a.smithi107.stderr: 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fd9b2952b43] 2023-12-04T05:45:37.307 INFO:tasks.ceph.mds.a.smithi107.stderr: 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fd9b29e4a00] 2023-12-04T05:45:37.308 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-12-04T05:45:37.303+0000 7fd9ace98640 -1 *** Caught signal (Aborted) ** 2023-12-04T05:45:37.308 INFO:tasks.ceph.mds.a.smithi107.stderr: in thread 7fd9ace98640 thread_name:ms_dispatch 2023-12-04T05:45:37.308 INFO:tasks.ceph.mds.a.smithi107.stderr: 2023-12-04T05:45:37.308 INFO:tasks.ceph.mds.a.smithi107.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:45:37.308 INFO:tasks.ceph.mds.a.smithi107.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fd9b2900520] 2023-12-04T05:45:37.308 INFO:tasks.ceph.mds.a.smithi107.stderr: 2: pthread_kill() 2023-12-04T05:45:37.308 INFO:tasks.ceph.mds.a.smithi107.stderr: 3: raise() 2023-12-04T05:45:37.308 INFO:tasks.ceph.mds.a.smithi107.stderr: 4: abort() 2023-12-04T05:45:37.308 INFO:tasks.ceph.mds.a.smithi107.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fd9b2fc5ef0] 2023-12-04T05:45:37.308 INFO:tasks.ceph.mds.a.smithi107.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7fd9b2fc604a] 2023-12-04T05:45:37.308 INFO:tasks.ceph.mds.a.smithi107.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x5636330593e8] 2023-12-04T05:45:37.308 INFO:tasks.ceph.mds.a.smithi107.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x56363304e174] 2023-12-04T05:45:37.308 INFO:tasks.ceph.mds.a.smithi107.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x563632fff033] 2023-12-04T05:45:37.308 INFO:tasks.ceph.mds.a.smithi107.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x563632f39dec] 2023-12-04T05:45:37.308 INFO:tasks.ceph.mds.a.smithi107.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x563632f40be8] 2023-12-04T05:45:37.309 INFO:tasks.ceph.mds.a.smithi107.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x563632f36a91] 2023-12-04T05:45:37.309 INFO:tasks.ceph.mds.a.smithi107.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x563632f3734c] 2023-12-04T05:45:37.309 INFO:tasks.ceph.mds.a.smithi107.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563632f23837] 2023-12-04T05:45:37.309 INFO:tasks.ceph.mds.a.smithi107.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x563632ea75fb] 2023-12-04T05:45:37.309 INFO:tasks.ceph.mds.a.smithi107.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563632ea7acc] 2023-12-04T05:45:37.309 INFO:tasks.ceph.mds.a.smithi107.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x563632e928e5] 2023-12-04T05:45:37.309 INFO:tasks.ceph.mds.a.smithi107.stderr: 18: (DispatchQueue::entry()+0x5ea) [0x7fd9b31b211a] 2023-12-04T05:45:37.309 INFO:tasks.ceph.mds.a.smithi107.stderr: 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7fd9b3244991] 2023-12-04T05:45:37.309 INFO:tasks.ceph.mds.a.smithi107.stderr: 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fd9b2952b43] 2023-12-04T05:45:37.309 INFO:tasks.ceph.mds.a.smithi107.stderr: 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fd9b29e4a00] 2023-12-04T05:45:37.309 INFO:tasks.ceph.mds.a.smithi107.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-04T05:45:37.309 INFO:tasks.ceph.mds.a.smithi107.stderr: 2023-12-04T05:45:37.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.319+0000 7fc00a5c2640 1 -- 172.21.15.107:0/253207629 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc00414b4c0 con 0x7fc004157180 2023-12-04T05:45:37.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.319+0000 7fc0017fa640 1 -- 172.21.15.107:0/253207629 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4972 (secure 0 0 0) 0x7fbffc054ab0 con 0x7fc004157180 2023-12-04T05:45:37.321 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:37.321 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:35.807482+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2,3],"up":{"mds_0":4288,"mds_1":5106,"mds_2":4263,"mds_3":4260},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":14,"state":"up:stopping","state_seq":9,"addr":"172.21.15.107:6836/3444851515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3444851515},{"type":"v1","addr":"172.21.15.107:6836","nonce":3444851515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":36,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6839/3319682203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":3319682203},{"type":"v1","addr":"172.21.15.107:6839","nonce":3319682203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:37.321 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 41 2023-12-04T05:45:37.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.323+0000 7fc00a5c2640 1 -- 172.21.15.107:0/253207629 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbfe40668d0 msgr2=0x7fbfe4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:37.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.323+0000 7fc00a5c2640 1 --2- 172.21.15.107:0/253207629 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbfe40668d0 0x7fbfe4068d90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbff40af000 tx=0x7fbff40a7000 comp rx=0 tx=0).stop 2023-12-04T05:45:37.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.323+0000 7fc00a5c2640 1 -- 172.21.15.107:0/253207629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc004157180 msgr2=0x7fc004151b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:37.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.323+0000 7fc00a5c2640 1 --2- 172.21.15.107:0/253207629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc004157180 0x7fc004151b90 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fbffc00b970 tx=0x7fbffc00be30 comp rx=0 tx=0).stop 2023-12-04T05:45:37.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.323+0000 7fc00a5c2640 1 -- 172.21.15.107:0/253207629 shutdown_connections 2023-12-04T05:45:37.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.323+0000 7fc00a5c2640 1 --2- 172.21.15.107:0/253207629 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbfe40668d0 0x7fbfe4068d90 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:37.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.323+0000 7fc00a5c2640 1 --2- 172.21.15.107:0/253207629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc004157180 0x7fc004151b90 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:37.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.323+0000 7fc00a5c2640 1 --2- 172.21.15.107:0/253207629 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc0040c4a80 0x7fc0041527d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:37.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.323+0000 7fc00a5c2640 1 --2- 172.21.15.107:0/253207629 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0040c46a0 0x7fc00414b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:37.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.323+0000 7fc00a5c2640 1 -- 172.21.15.107:0/253207629 >> 172.21.15.107:0/253207629 conn(0x7fc0040c7ff0 msgr2=0x7fc004153110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:37.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.323+0000 7fc00a5c2640 1 -- 172.21.15.107:0/253207629 shutdown_connections 2023-12-04T05:45:37.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:37.323+0000 7fc00a5c2640 1 -- 172.21.15.107:0/253207629 wait complete. 2023-12-04T05:45:37.331 INFO:tasks.ceph.mds.a.smithi107.stderr: -1> 2023-12-04T05:45:37.303+0000 7fd9ace98640 -1 ./src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7fd9ace98640 time 2023-12-04T05:45:37.302432+0000 2023-12-04T05:45:37.331 INFO:tasks.ceph.mds.a.smithi107.stderr:./src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-12-04T05:45:37.331 INFO:tasks.ceph.mds.a.smithi107.stderr: 2023-12-04T05:45:37.331 INFO:tasks.ceph.mds.a.smithi107.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:45:37.331 INFO:tasks.ceph.mds.a.smithi107.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7fd9b2fc5e96] 2023-12-04T05:45:37.331 INFO:tasks.ceph.mds.a.smithi107.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7fd9b2fc604a] 2023-12-04T05:45:37.331 INFO:tasks.ceph.mds.a.smithi107.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x5636330593e8] 2023-12-04T05:45:37.331 INFO:tasks.ceph.mds.a.smithi107.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x56363304e174] 2023-12-04T05:45:37.331 INFO:tasks.ceph.mds.a.smithi107.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x563632fff033] 2023-12-04T05:45:37.331 INFO:tasks.ceph.mds.a.smithi107.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x563632f39dec] 2023-12-04T05:45:37.331 INFO:tasks.ceph.mds.a.smithi107.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x563632f40be8] 2023-12-04T05:45:37.331 INFO:tasks.ceph.mds.a.smithi107.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x563632f36a91] 2023-12-04T05:45:37.331 INFO:tasks.ceph.mds.a.smithi107.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x563632f3734c] 2023-12-04T05:45:37.332 INFO:tasks.ceph.mds.a.smithi107.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563632f23837] 2023-12-04T05:45:37.332 INFO:tasks.ceph.mds.a.smithi107.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x563632ea75fb] 2023-12-04T05:45:37.332 INFO:tasks.ceph.mds.a.smithi107.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563632ea7acc] 2023-12-04T05:45:37.332 INFO:tasks.ceph.mds.a.smithi107.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x563632e928e5] 2023-12-04T05:45:37.332 INFO:tasks.ceph.mds.a.smithi107.stderr: 14: (DispatchQueue::entry()+0x5ea) [0x7fd9b31b211a] 2023-12-04T05:45:37.332 INFO:tasks.ceph.mds.a.smithi107.stderr: 15: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7fd9b3244991] 2023-12-04T05:45:37.332 INFO:tasks.ceph.mds.a.smithi107.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fd9b2952b43] 2023-12-04T05:45:37.332 INFO:tasks.ceph.mds.a.smithi107.stderr: 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fd9b29e4a00] 2023-12-04T05:45:37.332 INFO:tasks.ceph.mds.a.smithi107.stderr: 2023-12-04T05:45:37.332 INFO:tasks.ceph.mds.a.smithi107.stderr: 0> 2023-12-04T05:45:37.303+0000 7fd9ace98640 -1 *** Caught signal (Aborted) ** 2023-12-04T05:45:37.332 INFO:tasks.ceph.mds.a.smithi107.stderr: in thread 7fd9ace98640 thread_name:ms_dispatch 2023-12-04T05:45:37.332 INFO:tasks.ceph.mds.a.smithi107.stderr: 2023-12-04T05:45:37.332 INFO:tasks.ceph.mds.a.smithi107.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:45:37.332 INFO:tasks.ceph.mds.a.smithi107.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fd9b2900520] 2023-12-04T05:45:37.332 INFO:tasks.ceph.mds.a.smithi107.stderr: 2: pthread_kill() 2023-12-04T05:45:37.333 INFO:tasks.ceph.mds.a.smithi107.stderr: 3: raise() 2023-12-04T05:45:37.333 INFO:tasks.ceph.mds.a.smithi107.stderr: 4: abort() 2023-12-04T05:45:37.333 INFO:tasks.ceph.mds.a.smithi107.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fd9b2fc5ef0] 2023-12-04T05:45:37.333 INFO:tasks.ceph.mds.a.smithi107.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7fd9b2fc604a] 2023-12-04T05:45:37.333 INFO:tasks.ceph.mds.a.smithi107.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x5636330593e8] 2023-12-04T05:45:37.333 INFO:tasks.ceph.mds.a.smithi107.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x56363304e174] 2023-12-04T05:45:37.333 INFO:tasks.ceph.mds.a.smithi107.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x563632fff033] 2023-12-04T05:45:37.333 INFO:tasks.ceph.mds.a.smithi107.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x563632f39dec] 2023-12-04T05:45:37.333 INFO:tasks.ceph.mds.a.smithi107.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x563632f40be8] 2023-12-04T05:45:37.333 INFO:tasks.ceph.mds.a.smithi107.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x563632f36a91] 2023-12-04T05:45:37.333 INFO:tasks.ceph.mds.a.smithi107.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x563632f3734c] 2023-12-04T05:45:37.333 INFO:tasks.ceph.mds.a.smithi107.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563632f23837] 2023-12-04T05:45:37.333 INFO:tasks.ceph.mds.a.smithi107.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x563632ea75fb] 2023-12-04T05:45:37.334 INFO:tasks.ceph.mds.a.smithi107.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563632ea7acc] 2023-12-04T05:45:37.334 INFO:tasks.ceph.mds.a.smithi107.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x563632e928e5] 2023-12-04T05:45:37.334 INFO:tasks.ceph.mds.a.smithi107.stderr: 18: (DispatchQueue::entry()+0x5ea) [0x7fd9b31b211a] 2023-12-04T05:45:37.334 INFO:tasks.ceph.mds.a.smithi107.stderr: 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7fd9b3244991] 2023-12-04T05:45:37.334 INFO:tasks.ceph.mds.a.smithi107.stderr: 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fd9b2952b43] 2023-12-04T05:45:37.334 INFO:tasks.ceph.mds.a.smithi107.stderr: 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fd9b29e4a00] 2023-12-04T05:45:37.334 INFO:tasks.ceph.mds.a.smithi107.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-04T05:45:37.334 INFO:tasks.ceph.mds.a.smithi107.stderr: 2023-12-04T05:45:37.342 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:45:37.344 INFO:tasks.ceph.mds.a.smithi107.stderr: -1> 2023-12-04T05:45:37.303+0000 7fd9ace98640 -1 ./src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7fd9ace98640 time 2023-12-04T05:45:37.302432+0000 2023-12-04T05:45:37.344 INFO:tasks.ceph.mds.a.smithi107.stderr:./src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-12-04T05:45:37.344 INFO:tasks.ceph.mds.a.smithi107.stderr: 2023-12-04T05:45:37.344 INFO:tasks.ceph.mds.a.smithi107.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:45:37.344 INFO:tasks.ceph.mds.a.smithi107.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7fd9b2fc5e96] 2023-12-04T05:45:37.344 INFO:tasks.ceph.mds.a.smithi107.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7fd9b2fc604a] 2023-12-04T05:45:37.344 INFO:tasks.ceph.mds.a.smithi107.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x5636330593e8] 2023-12-04T05:45:37.344 INFO:tasks.ceph.mds.a.smithi107.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x56363304e174] 2023-12-04T05:45:37.344 INFO:tasks.ceph.mds.a.smithi107.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x563632fff033] 2023-12-04T05:45:37.344 INFO:tasks.ceph.mds.a.smithi107.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x563632f39dec] 2023-12-04T05:45:37.344 INFO:tasks.ceph.mds.a.smithi107.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x563632f40be8] 2023-12-04T05:45:37.344 INFO:tasks.ceph.mds.a.smithi107.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x563632f36a91] 2023-12-04T05:45:37.344 INFO:tasks.ceph.mds.a.smithi107.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x563632f3734c] 2023-12-04T05:45:37.344 INFO:tasks.ceph.mds.a.smithi107.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563632f23837] 2023-12-04T05:45:37.345 INFO:tasks.ceph.mds.a.smithi107.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x563632ea75fb] 2023-12-04T05:45:37.345 INFO:tasks.ceph.mds.a.smithi107.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563632ea7acc] 2023-12-04T05:45:37.345 INFO:tasks.ceph.mds.a.smithi107.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x563632e928e5] 2023-12-04T05:45:37.345 INFO:tasks.ceph.mds.a.smithi107.stderr: 14: (DispatchQueue::entry()+0x5ea) [0x7fd9b31b211a] 2023-12-04T05:45:37.345 INFO:tasks.ceph.mds.a.smithi107.stderr: 15: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7fd9b3244991] 2023-12-04T05:45:37.345 INFO:tasks.ceph.mds.a.smithi107.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fd9b2952b43] 2023-12-04T05:45:37.345 INFO:tasks.ceph.mds.a.smithi107.stderr: 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fd9b29e4a00] 2023-12-04T05:45:37.345 INFO:tasks.ceph.mds.a.smithi107.stderr: 2023-12-04T05:45:37.345 INFO:tasks.ceph.mds.a.smithi107.stderr: 0> 2023-12-04T05:45:37.303+0000 7fd9ace98640 -1 *** Caught signal (Aborted) ** 2023-12-04T05:45:37.345 INFO:tasks.ceph.mds.a.smithi107.stderr: in thread 7fd9ace98640 thread_name:ms_dispatch 2023-12-04T05:45:37.345 INFO:tasks.ceph.mds.a.smithi107.stderr: 2023-12-04T05:45:37.345 INFO:tasks.ceph.mds.a.smithi107.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-12-04T05:45:37.345 INFO:tasks.ceph.mds.a.smithi107.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fd9b2900520] 2023-12-04T05:45:37.345 INFO:tasks.ceph.mds.a.smithi107.stderr: 2: pthread_kill() 2023-12-04T05:45:37.345 INFO:tasks.ceph.mds.a.smithi107.stderr: 3: raise() 2023-12-04T05:45:37.346 INFO:tasks.ceph.mds.a.smithi107.stderr: 4: abort() 2023-12-04T05:45:37.346 INFO:tasks.ceph.mds.a.smithi107.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fd9b2fc5ef0] 2023-12-04T05:45:37.346 INFO:tasks.ceph.mds.a.smithi107.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7fd9b2fc604a] 2023-12-04T05:45:37.346 INFO:tasks.ceph.mds.a.smithi107.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x8d8) [0x5636330593e8] 2023-12-04T05:45:37.346 INFO:tasks.ceph.mds.a.smithi107.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20b4) [0x56363304e174] 2023-12-04T05:45:37.346 INFO:tasks.ceph.mds.a.smithi107.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x8d3) [0x563632fff033] 2023-12-04T05:45:37.346 INFO:tasks.ceph.mds.a.smithi107.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x21c) [0x563632f39dec] 2023-12-04T05:45:37.346 INFO:tasks.ceph.mds.a.smithi107.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x563632f40be8] 2023-12-04T05:45:37.346 INFO:tasks.ceph.mds.a.smithi107.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x801) [0x563632f36a91] 2023-12-04T05:45:37.346 INFO:tasks.ceph.mds.a.smithi107.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6ac) [0x563632f3734c] 2023-12-04T05:45:37.346 INFO:tasks.ceph.mds.a.smithi107.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563632f23837] 2023-12-04T05:45:37.346 INFO:tasks.ceph.mds.a.smithi107.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4cb) [0x563632ea75fb] 2023-12-04T05:45:37.346 INFO:tasks.ceph.mds.a.smithi107.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563632ea7acc] 2023-12-04T05:45:37.346 INFO:tasks.ceph.mds.a.smithi107.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x563632e928e5] 2023-12-04T05:45:37.346 INFO:tasks.ceph.mds.a.smithi107.stderr: 18: (DispatchQueue::entry()+0x5ea) [0x7fd9b31b211a] 2023-12-04T05:45:37.347 INFO:tasks.ceph.mds.a.smithi107.stderr: 19: /usr/lib/ceph/libceph-common.so.2(+0x3ee991) [0x7fd9b3244991] 2023-12-04T05:45:37.347 INFO:tasks.ceph.mds.a.smithi107.stderr: 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fd9b2952b43] 2023-12-04T05:45:37.347 INFO:tasks.ceph.mds.a.smithi107.stderr: 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fd9b29e4a00] 2023-12-04T05:45:37.347 INFO:tasks.ceph.mds.a.smithi107.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-04T05:45:37.347 INFO:tasks.ceph.mds.a.smithi107.stderr: 2023-12-04T05:45:37.678 INFO:tasks.ceph.mds.a.smithi107.stderr:daemon-helper: command crashed with signal 6 2023-12-04T05:45:38.998 INFO:tasks.ceph.mds.e.smithi107.stdout:starting mds.e at 2023-12-04T05:45:39.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:39.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.475+0000 7f771b53b640 1 Processor -- start 2023-12-04T05:45:39.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.475+0000 7f771b53b640 1 -- start start 2023-12-04T05:45:39.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.475+0000 7f771b53b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7714154290 0x7f771406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:39.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.475+0000 7f771b53b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771406e320 con 0x7f77140c6b70 2023-12-04T05:45:39.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.475+0000 7f771b53b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77140c66e0 con 0x7f77140c8c80 2023-12-04T05:45:39.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.475+0000 7f771b53b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7714071c10 con 0x7f7714154290 2023-12-04T05:45:39.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.475+0000 7f77192d7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7714154290 0x7f771406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:39.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.475+0000 7f7719ad8640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f77140c6b70 0x7f771415a710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36898/0 (socket says 172.21.15.107:36898) 2023-12-04T05:45:39.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.475+0000 7f7719ad8640 1 -- 172.21.15.107:0/1148224367 learned_addr learned my addr 172.21.15.107:0/1148224367 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:45:39.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.475+0000 7f77192d7640 1 -- 172.21.15.107:0/1148224367 >> v1:172.21.15.195:6789/0 conn(0x7f77140c8c80 legacy=0x7f771415a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:39.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.475+0000 7f77192d7640 1 -- 172.21.15.107:0/1148224367 >> v1:172.21.15.107:6789/0 conn(0x7f77140c6b70 legacy=0x7f771415a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:39.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.475+0000 7f77192d7640 1 -- 172.21.15.107:0/1148224367 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77140c6fc0 con 0x7f7714154290 2023-12-04T05:45:39.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.475+0000 7f770bfff640 1 -- 172.21.15.107:0/1148224367 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993531807 0 0) 0x7f771406e320 con 0x7f77140c6b70 2023-12-04T05:45:39.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.479+0000 7f77192d7640 1 --2- 172.21.15.107:0/1148224367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7714154290 0x7f771406fca0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f77100097c0 tx=0x7f771001c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feff657be6c5a18b server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:39.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.479+0000 7f770bfff640 1 -- 172.21.15.107:0/1148224367 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77100ab600 con 0x7f7714154290 2023-12-04T05:45:39.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.479+0000 7f770bfff640 1 -- 172.21.15.107:0/1148224367 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77100a7050 con 0x7f7714154290 2023-12-04T05:45:39.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.479+0000 7f770bfff640 1 -- 172.21.15.107:0/1148224367 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77100ab980 con 0x7f7714154290 2023-12-04T05:45:39.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.479+0000 7f771b53b640 1 -- 172.21.15.107:0/1148224367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7714154290 msgr2=0x7f771406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:39.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.479+0000 7f771b53b640 1 --2- 172.21.15.107:0/1148224367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7714154290 0x7f771406fca0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f77100097c0 tx=0x7f771001c320 comp rx=0 tx=0).stop 2023-12-04T05:45:39.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.479+0000 7f771b53b640 1 -- 172.21.15.107:0/1148224367 shutdown_connections 2023-12-04T05:45:39.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.479+0000 7f771b53b640 1 --2- 172.21.15.107:0/1148224367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7714154290 0x7f771406fca0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:39.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.479+0000 7f771b53b640 1 -- 172.21.15.107:0/1148224367 >> 172.21.15.107:0/1148224367 conn(0x7f77140c7ff0 msgr2=0x7f77140c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:39.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.479+0000 7f771b53b640 1 -- 172.21.15.107:0/1148224367 shutdown_connections 2023-12-04T05:45:39.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.479+0000 7f771b53b640 1 -- 172.21.15.107:0/1148224367 wait complete. 2023-12-04T05:45:39.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.479+0000 7f771b53b640 1 Processor -- start 2023-12-04T05:45:39.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.479+0000 7f771b53b640 1 -- start start 2023-12-04T05:45:39.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.479+0000 7f771b53b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77140c6b70 0x7f771413d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:39.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.479+0000 7f771b53b640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f77140c8c80 0x7f7714144110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:39.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.479+0000 7f771b53b640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7714154290 0x7f7714144650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:39.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.479+0000 7f771b53b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7714159350 con 0x7f77140c6b70 2023-12-04T05:45:39.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.479+0000 7f771b53b640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77141f33a0 con 0x7f7714154290 2023-12-04T05:45:39.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.479+0000 7f771b53b640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7714143530 con 0x7f77140c8c80 2023-12-04T05:45:39.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.479+0000 7f77192d7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77140c6b70 0x7f771413d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:39.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.479+0000 7f77192d7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77140c6b70 0x7f771413d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37214/0 (socket says 172.21.15.107:37214) 2023-12-04T05:45:39.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.479+0000 7f77192d7640 1 -- 172.21.15.107:0/4201237987 learned_addr learned my addr 172.21.15.107:0/4201237987 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:39.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.483+0000 7f7719ad8640 1 --2- 172.21.15.107:0/4201237987 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7714154290 0x7f7714144650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:39.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.483+0000 7f77192d7640 1 -- 172.21.15.107:0/4201237987 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f77140c8c80 msgr2=0x7f7714144110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:39.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.483+0000 7f7718ad6640 1 --2- 172.21.15.107:0/4201237987 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f77140c8c80 0x7f7714144110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:39.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.483+0000 7f77192d7640 1 --2- 172.21.15.107:0/4201237987 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f77140c8c80 0x7f7714144110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:39.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.483+0000 7f77192d7640 1 -- 172.21.15.107:0/4201237987 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7714154290 msgr2=0x7f7714144650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:39.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.483+0000 7f77192d7640 1 --2- 172.21.15.107:0/4201237987 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7714154290 0x7f7714144650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:39.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.483+0000 7f77192d7640 1 -- 172.21.15.107:0/4201237987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7710009420 con 0x7f77140c6b70 2023-12-04T05:45:39.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.483+0000 7f7719ad8640 1 --2- 172.21.15.107:0/4201237987 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7714154290 0x7f7714144650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:45:39.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.483+0000 7f77192d7640 1 --2- 172.21.15.107:0/4201237987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77140c6b70 0x7f771413d800 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f77100ae040 tx=0x7f771001ab60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:39.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.483+0000 7f7709ffb640 1 -- 172.21.15.107:0/4201237987 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77100b7650 con 0x7f77140c6b70 2023-12-04T05:45:39.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.483+0000 7f7709ffb640 1 -- 172.21.15.107:0/4201237987 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77100b77d0 con 0x7f77140c6b70 2023-12-04T05:45:39.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.483+0000 7f771b53b640 1 -- 172.21.15.107:0/4201237987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77141f27b0 con 0x7f77140c6b70 2023-12-04T05:45:39.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.483+0000 7f7709ffb640 1 -- 172.21.15.107:0/4201237987 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77100b7b80 con 0x7f77140c6b70 2023-12-04T05:45:39.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.483+0000 7f771b53b640 1 -- 172.21.15.107:0/4201237987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77141f2a70 con 0x7f77140c6b70 2023-12-04T05:45:39.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.483+0000 7f771b53b640 1 -- 172.21.15.107:0/4201237987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7714067870 con 0x7f77140c6b70 2023-12-04T05:45:39.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.483+0000 7f7709ffb640 1 -- 172.21.15.107:0/4201237987 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f77100b7ce0 con 0x7f77140c6b70 2023-12-04T05:45:39.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.483+0000 7f7709ffb640 1 --2- 172.21.15.107:0/4201237987 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f76f80668d0 0x7f76f8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:39.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.483+0000 7f7709ffb640 1 -- 172.21.15.107:0/4201237987 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f771011f000 con 0x7f77140c6b70 2023-12-04T05:45:39.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.483+0000 7f7718ad6640 1 --2- 172.21.15.107:0/4201237987 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f76f80668d0 0x7f76f8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:39.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.483+0000 7f7718ad6640 1 --2- 172.21.15.107:0/4201237987 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f76f80668d0 0x7f76f8068d90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7704007fd0 tx=0x7f7704009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:39.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.487+0000 7f7709ffb640 1 -- 172.21.15.107:0/4201237987 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f77100ec150 con 0x7f77140c6b70 2023-12-04T05:45:39.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.671+0000 7f771b53b640 1 -- 172.21.15.107:0/4201237987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77140c7850 con 0x7f77140c6b70 2023-12-04T05:45:39.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.671+0000 7f7709ffb640 1 -- 172.21.15.107:0/4201237987 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4166 (secure 0 0 0) 0x7f77100b0090 con 0x7f77140c6b70 2023-12-04T05:45:39.672 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:39.672 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:38.916494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4288,"mds_1":5106,"mds_2":4263},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:stopping","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":36,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6839/3319682203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":3319682203},{"type":"v1","addr":"172.21.15.107:6839","nonce":3319682203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:39.672 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 43 2023-12-04T05:45:39.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.671+0000 7f771b53b640 1 -- 172.21.15.107:0/4201237987 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f76f80668d0 msgr2=0x7f76f8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:39.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.671+0000 7f771b53b640 1 --2- 172.21.15.107:0/4201237987 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f76f80668d0 0x7f76f8068d90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7704007fd0 tx=0x7f7704009040 comp rx=0 tx=0).stop 2023-12-04T05:45:39.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.671+0000 7f771b53b640 1 -- 172.21.15.107:0/4201237987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77140c6b70 msgr2=0x7f771413d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:39.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.671+0000 7f771b53b640 1 --2- 172.21.15.107:0/4201237987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77140c6b70 0x7f771413d800 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f77100ae040 tx=0x7f771001ab60 comp rx=0 tx=0).stop 2023-12-04T05:45:39.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.671+0000 7f771b53b640 1 -- 172.21.15.107:0/4201237987 shutdown_connections 2023-12-04T05:45:39.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.671+0000 7f771b53b640 1 --2- 172.21.15.107:0/4201237987 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f76f80668d0 0x7f76f8068d90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:39.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.671+0000 7f771b53b640 1 --2- 172.21.15.107:0/4201237987 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7714154290 0x7f7714144650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:39.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.671+0000 7f771b53b640 1 --2- 172.21.15.107:0/4201237987 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f77140c8c80 0x7f7714144110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:39.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.671+0000 7f771b53b640 1 --2- 172.21.15.107:0/4201237987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77140c6b70 0x7f771413d800 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:39.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.671+0000 7f771b53b640 1 -- 172.21.15.107:0/4201237987 >> 172.21.15.107:0/4201237987 conn(0x7f77140c7ff0 msgr2=0x7f771415a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:39.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.671+0000 7f771b53b640 1 -- 172.21.15.107:0/4201237987 shutdown_connections 2023-12-04T05:45:39.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:39.671+0000 7f771b53b640 1 -- 172.21.15.107:0/4201237987 wait complete. 2023-12-04T05:45:39.690 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:45:41.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:41.745 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2023-12-04T05:45:41.745 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~114s 2023-12-04T05:45:41.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~155s 2023-12-04T05:45:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.819+0000 7f5d66407640 1 Processor -- start 2023-12-04T05:45:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.819+0000 7f5d66407640 1 -- start start 2023-12-04T05:45:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.819+0000 7f5d66407640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5d60072110 0x7f5d60158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.819+0000 7f5d66407640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5d600724f0 0x7f5d600c7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.819+0000 7f5d66407640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d60158da0 con 0x7f5d600c56f0 2023-12-04T05:45:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.819+0000 7f5d66407640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d601583d0 con 0x7f5d60072110 2023-12-04T05:45:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.819+0000 7f5d66407640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d600c6170 con 0x7f5d600724f0 2023-12-04T05:45:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.819+0000 7f5d5ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5d60072110 0x7f5d60158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.819+0000 7f5d5ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5d60072110 0x7f5d60158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37224/0 (socket says 172.21.15.107:37224) 2023-12-04T05:45:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.819+0000 7f5d5ffff640 1 -- 172.21.15.107:0/3893105248 learned_addr learned my addr 172.21.15.107:0/3893105248 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.819+0000 7f5d57fff640 1 --2- 172.21.15.107:0/3893105248 >> v2:172.21.15.195:3300/0 conn(0x7f5d600724f0 0x7f5d600c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.819+0000 7f5d5ffff640 1 -- 172.21.15.107:0/3893105248 >> v2:172.21.15.195:3300/0 conn(0x7f5d600724f0 msgr2=0x7f5d600c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.819+0000 7f5d5ffff640 1 --2- 172.21.15.107:0/3893105248 >> v2:172.21.15.195:3300/0 conn(0x7f5d600724f0 0x7f5d600c7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:41.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.819+0000 7f5d5ffff640 1 -- 172.21.15.107:0/3893105248 >> v1:172.21.15.195:6789/0 conn(0x7f5d600c56f0 legacy=0x7f5d60140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:41.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.819+0000 7f5d5ffff640 1 -- 172.21.15.107:0/3893105248 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d60157da0 con 0x7f5d60072110 2023-12-04T05:45:41.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.819+0000 7f5d57fff640 1 --2- 172.21.15.107:0/3893105248 >> v2:172.21.15.195:3300/0 conn(0x7f5d600724f0 0x7f5d600c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:45:41.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d5ffff640 1 --2- 172.21.15.107:0/3893105248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d60072110 0x7f5d60158860 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5d6006b870 tx=0x7f5d5001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ce4a22f1916f896 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:41.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d5f7fe640 1 -- 172.21.15.107:0/3893105248 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d500a89f0 con 0x7f5d60072110 2023-12-04T05:45:41.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d5f7fe640 1 -- 172.21.15.107:0/3893105248 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d500a8b50 con 0x7f5d60072110 2023-12-04T05:45:41.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d5f7fe640 1 -- 172.21.15.107:0/3893105248 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d500b0440 con 0x7f5d60072110 2023-12-04T05:45:41.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d66407640 1 -- 172.21.15.107:0/3893105248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d60072110 msgr2=0x7f5d60158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:41.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d66407640 1 --2- 172.21.15.107:0/3893105248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d60072110 0x7f5d60158860 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5d6006b870 tx=0x7f5d5001ecf0 comp rx=0 tx=0).stop 2023-12-04T05:45:41.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d66407640 1 -- 172.21.15.107:0/3893105248 shutdown_connections 2023-12-04T05:45:41.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d66407640 1 --2- 172.21.15.107:0/3893105248 >> v2:172.21.15.195:3300/0 conn(0x7f5d600724f0 0x7f5d600c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:41.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d66407640 1 --2- 172.21.15.107:0/3893105248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d60072110 0x7f5d60158860 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:41.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d66407640 1 -- 172.21.15.107:0/3893105248 >> 172.21.15.107:0/3893105248 conn(0x7f5d600c7ff0 msgr2=0x7f5d600c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:41.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d66407640 1 -- 172.21.15.107:0/3893105248 shutdown_connections 2023-12-04T05:45:41.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d66407640 1 -- 172.21.15.107:0/3893105248 wait complete. 2023-12-04T05:45:41.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d66407640 1 Processor -- start 2023-12-04T05:45:41.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d66407640 1 -- start start 2023-12-04T05:45:41.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d66407640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d60072110 0x7f5d601496c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:41.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d66407640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d600724f0 0x7f5d601f3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:41.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d66407640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5d600c56f0 0x7f5d60147150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:41.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d66407640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6014acc0 con 0x7f5d600724f0 2023-12-04T05:45:41.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d66407640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d601595c0 con 0x7f5d60072110 2023-12-04T05:45:41.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d66407640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d601f3cb0 con 0x7f5d600c56f0 2023-12-04T05:45:41.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d57fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d600724f0 0x7f5d601f3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:41.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d57fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d600724f0 0x7f5d601f3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37230/0 (socket says 172.21.15.107:37230) 2023-12-04T05:45:41.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d57fff640 1 -- 172.21.15.107:0/1975974673 learned_addr learned my addr 172.21.15.107:0/1975974673 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:41.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d649a4640 1 --2- 172.21.15.107:0/1975974673 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5d600c56f0 0x7f5d60147150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:41.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d5ffff640 1 --2- 172.21.15.107:0/1975974673 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d60072110 0x7f5d601496c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:41.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d57fff640 1 -- 172.21.15.107:0/1975974673 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5d600c56f0 msgr2=0x7f5d60147150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:41.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d57fff640 1 --2- 172.21.15.107:0/1975974673 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5d600c56f0 0x7f5d60147150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d57fff640 1 -- 172.21.15.107:0/1975974673 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d60072110 msgr2=0x7f5d601496c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d57fff640 1 --2- 172.21.15.107:0/1975974673 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d60072110 0x7f5d601496c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d57fff640 1 -- 172.21.15.107:0/1975974673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d500094d0 con 0x7f5d600724f0 2023-12-04T05:45:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d5ffff640 1 --2- 172.21.15.107:0/1975974673 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d60072110 0x7f5d601496c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:45:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d57fff640 1 --2- 172.21.15.107:0/1975974673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d600724f0 0x7f5d601f3770 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5d4800c7f0 tx=0x7f5d4800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d5d7fa640 1 -- 172.21.15.107:0/1975974673 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4800cea0 con 0x7f5d600724f0 2023-12-04T05:45:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d5d7fa640 1 -- 172.21.15.107:0/1975974673 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d48007a30 con 0x7f5d600724f0 2023-12-04T05:45:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d5d7fa640 1 -- 172.21.15.107:0/1975974673 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d48007de0 con 0x7f5d600724f0 2023-12-04T05:45:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d66407640 1 -- 172.21.15.107:0/1975974673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d6014af10 con 0x7f5d600724f0 2023-12-04T05:45:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.823+0000 7f5d66407640 1 -- 172.21.15.107:0/1975974673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d601f2f70 con 0x7f5d600724f0 2023-12-04T05:45:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.827+0000 7f5d66407640 1 -- 172.21.15.107:0/1975974673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d6014b430 con 0x7f5d600724f0 2023-12-04T05:45:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.827+0000 7f5d5d7fa640 1 -- 172.21.15.107:0/1975974673 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f5d480113f0 con 0x7f5d600724f0 2023-12-04T05:45:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.827+0000 7f5d5d7fa640 1 --2- 172.21.15.107:0/1975974673 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5d400668d0 0x7f5d40068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.827+0000 7f5d5d7fa640 1 -- 172.21.15.107:0/1975974673 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f5d48084690 con 0x7f5d600724f0 2023-12-04T05:45:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.827+0000 7f5d5ffff640 1 --2- 172.21.15.107:0/1975974673 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5d400668d0 0x7f5d40068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:41.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.827+0000 7f5d5ffff640 1 --2- 172.21.15.107:0/1975974673 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5d400668d0 0x7f5d40068d90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5d500b2000 tx=0x7f5d500aa070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:41.831+0000 7f5d5d7fa640 1 -- 172.21.15.107:0/1975974673 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5d6014b430 con 0x7f5d600724f0 2023-12-04T05:45:42.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:42.015+0000 7f5d66407640 1 -- 172.21.15.107:0/1975974673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d6014b430 con 0x7f5d600724f0 2023-12-04T05:45:42.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:42.015+0000 7f5d5d7fa640 1 -- 172.21.15.107:0/1975974673 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4959 (secure 0 0 0) 0x7f5d6014b430 con 0x7f5d600724f0 2023-12-04T05:45:42.017 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:42.017 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5190,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:38.916494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4288,"mds_1":5106,"mds_2":4263},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:stopping","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":36,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6839/3319682203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":3319682203},{"type":"v1","addr":"172.21.15.107:6839","nonce":3319682203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:42.017 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 44 2023-12-04T05:45:42.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:42.019+0000 7f5d66407640 1 -- 172.21.15.107:0/1975974673 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5d400668d0 msgr2=0x7f5d40068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:42.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:42.019+0000 7f5d66407640 1 --2- 172.21.15.107:0/1975974673 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5d400668d0 0x7f5d40068d90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5d500b2000 tx=0x7f5d500aa070 comp rx=0 tx=0).stop 2023-12-04T05:45:42.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:42.019+0000 7f5d66407640 1 -- 172.21.15.107:0/1975974673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d600724f0 msgr2=0x7f5d601f3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:42.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:42.019+0000 7f5d66407640 1 --2- 172.21.15.107:0/1975974673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d600724f0 0x7f5d601f3770 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5d4800c7f0 tx=0x7f5d4800ccb0 comp rx=0 tx=0).stop 2023-12-04T05:45:42.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:42.019+0000 7f5d66407640 1 -- 172.21.15.107:0/1975974673 shutdown_connections 2023-12-04T05:45:42.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:42.019+0000 7f5d66407640 1 --2- 172.21.15.107:0/1975974673 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5d400668d0 0x7f5d40068d90 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:42.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:42.019+0000 7f5d66407640 1 --2- 172.21.15.107:0/1975974673 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5d600c56f0 0x7f5d60147150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:42.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:42.019+0000 7f5d66407640 1 --2- 172.21.15.107:0/1975974673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d600724f0 0x7f5d601f3770 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:42.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:42.019+0000 7f5d66407640 1 --2- 172.21.15.107:0/1975974673 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d60072110 0x7f5d601496c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:42.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:42.019+0000 7f5d66407640 1 -- 172.21.15.107:0/1975974673 >> 172.21.15.107:0/1975974673 conn(0x7f5d600c7ff0 msgr2=0x7f5d6006d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:42.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:42.019+0000 7f5d66407640 1 -- 172.21.15.107:0/1975974673 shutdown_connections 2023-12-04T05:45:42.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:42.019+0000 7f5d66407640 1 -- 172.21.15.107:0/1975974673 wait complete. 2023-12-04T05:45:42.034 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-12-04T05:45:42.035 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 44, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-12-04T05:39:59.032019+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 43, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4263": { "addr": "172.21.15.107:6837/2210527429", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6835", "nonce": 2210527429, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 2210527429, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4263, "incarnation": 12, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:stopping", "state_seq": 7 }, "gid_4288": { "addr": "172.21.15.195:6837/3412865477", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6835", "nonce": 3412865477, "type": "v2" }, { "addr": "172.21.15.195:6837", "nonce": 3412865477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_5106": { "addr": "172.21.15.107:6839/3319682203", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6838", "nonce": 3319682203, "type": "v2" }, { "addr": "172.21.15.107:6839", "nonce": 3319682203, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5106, "incarnation": 36, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 26, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-12-04T05:45:38.916494+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 3, 4 ], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 5106, "mds_2": 4263 } } } ], "standbys": [ { "addr": "172.21.15.107:6836/2928623624", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 2928623624, "type": "v2" }, { "addr": "172.21.15.107:6836", "nonce": 2928623624, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 44, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5190, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-12-04T05:45:44.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:44.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.167+0000 7f4dd6158640 1 Processor -- start 2023-12-04T05:45:44.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.167+0000 7f4dd6158640 1 -- start start 2023-12-04T05:45:44.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.167+0000 7f4dd6158640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4dd00c9850 0x7f4dd00cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:44.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.167+0000 7f4dd6158640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd0157e60 con 0x7f4dd0151fa0 2023-12-04T05:45:44.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.167+0000 7f4dd6158640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd0158140 con 0x7f4dd00cd9e0 2023-12-04T05:45:44.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.167+0000 7f4dd6158640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd00cddf0 con 0x7f4dd00c9850 2023-12-04T05:45:44.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.167+0000 7f4dcf7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4dd0151fa0 0x7f4dd0072520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36908/0 (socket says 172.21.15.107:36908) 2023-12-04T05:45:44.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.167+0000 7f4dcf7fe640 1 -- 172.21.15.107:0/1469640654 learned_addr learned my addr 172.21.15.107:0/1469640654 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:45:44.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.167+0000 7f4dceffd640 1 --2- 172.21.15.107:0/1469640654 >> v2:172.21.15.107:3300/0 conn(0x7f4dd00c9850 0x7f4dd00cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:44.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.167+0000 7f4dce7fc640 1 -- 172.21.15.107:0/1469640654 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742214194 0 0) 0x7f4dd0157e60 con 0x7f4dd0151fa0 2023-12-04T05:45:44.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.167+0000 7f4dce7fc640 1 -- 172.21.15.107:0/1469640654 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dbc003530 con 0x7f4dd0151fa0 2023-12-04T05:45:44.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.167+0000 7f4dceffd640 1 -- 172.21.15.107:0/1469640654 >> v1:172.21.15.195:6789/0 conn(0x7f4dd00cd9e0 legacy=0x7f4dd015b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:44.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.167+0000 7f4dceffd640 1 -- 172.21.15.107:0/1469640654 >> v1:172.21.15.107:6789/0 conn(0x7f4dd0151fa0 legacy=0x7f4dd0072520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:44.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.167+0000 7f4dceffd640 1 -- 172.21.15.107:0/1469640654 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd0158420 con 0x7f4dd00c9850 2023-12-04T05:45:44.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.171+0000 7f4dceffd640 1 --2- 172.21.15.107:0/1469640654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dd00c9850 0x7f4dd00cd4a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0009d90 tx=0x7f4dc001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e81e7afcb41574 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:44.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.171+0000 7f4dce7fc640 1 -- 172.21.15.107:0/1469640654 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc00a59d0 con 0x7f4dd00c9850 2023-12-04T05:45:44.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.171+0000 7f4dce7fc640 1 -- 172.21.15.107:0/1469640654 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc00a5b30 con 0x7f4dd00c9850 2023-12-04T05:45:44.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.171+0000 7f4dce7fc640 1 -- 172.21.15.107:0/1469640654 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc00ad440 con 0x7f4dd00c9850 2023-12-04T05:45:44.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.171+0000 7f4dd6158640 1 -- 172.21.15.107:0/1469640654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dd00c9850 msgr2=0x7f4dd00cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:44.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.171+0000 7f4dd6158640 1 --2- 172.21.15.107:0/1469640654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dd00c9850 0x7f4dd00cd4a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0009d90 tx=0x7f4dc001c910 comp rx=0 tx=0).stop 2023-12-04T05:45:44.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.171+0000 7f4dd6158640 1 -- 172.21.15.107:0/1469640654 shutdown_connections 2023-12-04T05:45:44.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.171+0000 7f4dd6158640 1 --2- 172.21.15.107:0/1469640654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dd00c9850 0x7f4dd00cd4a0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:44.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.171+0000 7f4dd6158640 1 -- 172.21.15.107:0/1469640654 >> 172.21.15.107:0/1469640654 conn(0x7f4dd00663a0 msgr2=0x7f4dd00667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:44.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.171+0000 7f4dd6158640 1 -- 172.21.15.107:0/1469640654 shutdown_connections 2023-12-04T05:45:44.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.171+0000 7f4dd6158640 1 -- 172.21.15.107:0/1469640654 wait complete. 2023-12-04T05:45:44.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.171+0000 7f4dd6158640 1 Processor -- start 2023-12-04T05:45:44.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.171+0000 7f4dd6158640 1 -- start start 2023-12-04T05:45:44.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dd6158640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dd00c9850 0x7f4dd01565f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:44.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dd6158640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4dd00cd9e0 0x7f4dd014a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:44.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dd6158640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4dd0151fa0 0x7f4dd014a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:44.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dd6158640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd014bd20 con 0x7f4dd00c9850 2023-12-04T05:45:44.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dd6158640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd0146580 con 0x7f4dd0151fa0 2023-12-04T05:45:44.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dd6158640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd014b1a0 con 0x7f4dd00cd9e0 2023-12-04T05:45:44.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dcf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dd00c9850 0x7f4dd01565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:44.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dcf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dd00c9850 0x7f4dd01565f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37244/0 (socket says 172.21.15.107:37244) 2023-12-04T05:45:44.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dcf7fe640 1 -- 172.21.15.107:0/2920049553 learned_addr learned my addr 172.21.15.107:0/2920049553 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:44.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dceffd640 1 --2- 172.21.15.107:0/2920049553 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4dd00cd9e0 0x7f4dd014a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:44.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dcffff640 1 --2- 172.21.15.107:0/2920049553 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4dd0151fa0 0x7f4dd014a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:44.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dcf7fe640 1 -- 172.21.15.107:0/2920049553 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4dd00cd9e0 msgr2=0x7f4dd014a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:44.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dcf7fe640 1 --2- 172.21.15.107:0/2920049553 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4dd00cd9e0 0x7f4dd014a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:44.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dcf7fe640 1 -- 172.21.15.107:0/2920049553 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4dd0151fa0 msgr2=0x7f4dd014a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:44.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dcf7fe640 1 --2- 172.21.15.107:0/2920049553 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4dd0151fa0 0x7f4dd014a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:44.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dcf7fe640 1 -- 172.21.15.107:0/2920049553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc0009a70 con 0x7f4dd00c9850 2023-12-04T05:45:44.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dcf7fe640 1 --2- 172.21.15.107:0/2920049553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dd00c9850 0x7f4dd01565f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4dc800c420 tx=0x7f4dc800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:44.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dc7fff640 1 -- 172.21.15.107:0/2920049553 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc8005cf0 con 0x7f4dd00c9850 2023-12-04T05:45:44.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dc7fff640 1 -- 172.21.15.107:0/2920049553 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc8005e50 con 0x7f4dd00c9850 2023-12-04T05:45:44.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dd6158640 1 -- 172.21.15.107:0/2920049553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd014b480 con 0x7f4dd00c9850 2023-12-04T05:45:44.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dc7fff640 1 -- 172.21.15.107:0/2920049553 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc80072d0 con 0x7f4dd00c9850 2023-12-04T05:45:44.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dd6158640 1 -- 172.21.15.107:0/2920049553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd014ba40 con 0x7f4dd00c9850 2023-12-04T05:45:44.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dd6158640 1 -- 172.21.15.107:0/2920049553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd0067ec0 con 0x7f4dd00c9850 2023-12-04T05:45:44.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dc7fff640 1 -- 172.21.15.107:0/2920049553 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4dc8007430 con 0x7f4dd00c9850 2023-12-04T05:45:44.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.175+0000 7f4dc7fff640 1 --2- 172.21.15.107:0/2920049553 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4db00668d0 0x7f4db0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:44.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.179+0000 7f4dc7fff640 1 -- 172.21.15.107:0/2920049553 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f4dc8083e30 con 0x7f4dd00c9850 2023-12-04T05:45:44.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.179+0000 7f4dceffd640 1 --2- 172.21.15.107:0/2920049553 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4db00668d0 0x7f4db0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:44.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.179+0000 7f4dceffd640 1 --2- 172.21.15.107:0/2920049553 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4db00668d0 0x7f4db0068d90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00af000 tx=0x7f4dc00a7070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:44.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.179+0000 7f4dc7fff640 1 -- 172.21.15.107:0/2920049553 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4dc8051110 con 0x7f4dd00c9850 2023-12-04T05:45:44.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.367+0000 7f4dd6158640 1 -- 172.21.15.107:0/2920049553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4dd0146e00 con 0x7f4dd00c9850 2023-12-04T05:45:44.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.367+0000 7f4dc7fff640 1 -- 172.21.15.107:0/2920049553 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4959 (secure 0 0 0) 0x7f4dc8021090 con 0x7f4dd00c9850 2023-12-04T05:45:44.368 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:44.369 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5190,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:38.916494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4288,"mds_1":5106,"mds_2":4263},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:stopping","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":36,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6839/3319682203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":3319682203},{"type":"v1","addr":"172.21.15.107:6839","nonce":3319682203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:44.369 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 44 2023-12-04T05:45:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.367+0000 7f4dd6158640 1 -- 172.21.15.107:0/2920049553 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4db00668d0 msgr2=0x7f4db0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.367+0000 7f4dd6158640 1 --2- 172.21.15.107:0/2920049553 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4db00668d0 0x7f4db0068d90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00af000 tx=0x7f4dc00a7070 comp rx=0 tx=0).stop 2023-12-04T05:45:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.367+0000 7f4dd6158640 1 -- 172.21.15.107:0/2920049553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dd00c9850 msgr2=0x7f4dd01565f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.367+0000 7f4dd6158640 1 --2- 172.21.15.107:0/2920049553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dd00c9850 0x7f4dd01565f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4dc800c420 tx=0x7f4dc800c8e0 comp rx=0 tx=0).stop 2023-12-04T05:45:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.367+0000 7f4dd6158640 1 -- 172.21.15.107:0/2920049553 shutdown_connections 2023-12-04T05:45:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.367+0000 7f4dd6158640 1 --2- 172.21.15.107:0/2920049553 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4db00668d0 0x7f4db0068d90 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.367+0000 7f4dd6158640 1 --2- 172.21.15.107:0/2920049553 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4dd0151fa0 0x7f4dd014a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.367+0000 7f4dd6158640 1 --2- 172.21.15.107:0/2920049553 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4dd00cd9e0 0x7f4dd014a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.367+0000 7f4dd6158640 1 --2- 172.21.15.107:0/2920049553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dd00c9850 0x7f4dd01565f0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.367+0000 7f4dd6158640 1 -- 172.21.15.107:0/2920049553 >> 172.21.15.107:0/2920049553 conn(0x7f4dd00663a0 msgr2=0x7f4dd0148300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.367+0000 7f4dd6158640 1 -- 172.21.15.107:0/2920049553 shutdown_connections 2023-12-04T05:45:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:44.367+0000 7f4dd6158640 1 -- 172.21.15.107:0/2920049553 wait complete. 2023-12-04T05:45:44.388 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:45:46.388 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:46.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.515+0000 7f9cd8dd1640 1 Processor -- start 2023-12-04T05:45:46.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd8dd1640 1 -- start start 2023-12-04T05:45:46.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd8dd1640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9cd4072110 0x7f9cd4158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:46.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd8dd1640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9cd40724f0 0x7f9cd40c7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:46.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd8dd1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cd4158da0 con 0x7f9cd40c56f0 2023-12-04T05:45:46.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd8dd1640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cd41583d0 con 0x7f9cd40724f0 2023-12-04T05:45:46.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd8dd1640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd40c6170 con 0x7f9cd4072110 2023-12-04T05:45:46.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd2d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9cd40c56f0 0x7f9cd4140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36922/0 (socket says 172.21.15.107:36922) 2023-12-04T05:45:46.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd2d9d640 1 -- 172.21.15.107:0/459758239 learned_addr learned my addr 172.21.15.107:0/459758239 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:45:46.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd159a640 1 -- 172.21.15.107:0/459758239 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2118348536 0 0) 0x7f9ccc0014a0 con 0x7f9cd40c56f0 2023-12-04T05:45:46.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd1d9b640 1 --2- 172.21.15.107:0/459758239 >> v2:172.21.15.195:3300/0 conn(0x7f9cd40724f0 0x7f9cd40c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:46.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd159a640 1 -- 172.21.15.107:0/459758239 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cbc003530 con 0x7f9cd40c56f0 2023-12-04T05:45:46.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd259c640 1 --2- 172.21.15.107:0/459758239 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9cd4072110 0x7f9cd4158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:46.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd159a640 1 -- 172.21.15.107:0/459758239 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2166510429 0 0) 0x7f9cbc003530 con 0x7f9cd40c56f0 2023-12-04T05:45:46.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd159a640 1 -- 172.21.15.107:0/459758239 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9cd4072110 msgr2=0x7f9cd4158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:46.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd159a640 1 --2- 172.21.15.107:0/459758239 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9cd4072110 0x7f9cd4158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:46.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd159a640 1 -- 172.21.15.107:0/459758239 >> v2:172.21.15.195:3300/0 conn(0x7f9cd40724f0 msgr2=0x7f9cd40c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:46.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd159a640 1 --2- 172.21.15.107:0/459758239 >> v2:172.21.15.195:3300/0 conn(0x7f9cd40724f0 0x7f9cd40c7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:46.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd159a640 1 -- 172.21.15.107:0/459758239 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd4157da0 con 0x7f9cd40c56f0 2023-12-04T05:45:46.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd159a640 1 -- 172.21.15.107:0/459758239 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f9ccc001b90 con 0x7f9cd40c56f0 2023-12-04T05:45:46.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd159a640 1 -- 172.21.15.107:0/459758239 >> v1:172.21.15.107:6789/0 conn(0x7f9cd40c56f0 legacy=0x7f9cd4140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:46.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd159a640 1 --2- 172.21.15.107:0/459758239 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9cd4072110 0x7f9cbc003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:46.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd159a640 1 --2- 172.21.15.107:0/459758239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cbc003fb0 0x7f9cbc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:46.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd159a640 1 --2- 172.21.15.107:0/459758239 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9cbc004d10 0x7f9cbc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:46.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd159a640 1 -- 172.21.15.107:0/459758239 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd41583d0 con 0x7f9cbc003fb0 2023-12-04T05:45:46.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd159a640 1 -- 172.21.15.107:0/459758239 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd40c6170 con 0x7f9cbc004d10 2023-12-04T05:45:46.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd159a640 1 -- 172.21.15.107:0/459758239 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cbc00ba90 con 0x7f9cd4072110 2023-12-04T05:45:46.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd1d9b640 1 --2- 172.21.15.107:0/459758239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cbc003fb0 0x7f9cbc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:46.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd2d9d640 1 --2- 172.21.15.107:0/459758239 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9cbc004d10 0x7f9cbc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:46.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd259c640 1 --2- 172.21.15.107:0/459758239 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9cd4072110 0x7f9cbc003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:46.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd1d9b640 1 -- 172.21.15.107:0/459758239 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9cd4072110 msgr2=0x7f9cbc003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:46.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd1d9b640 1 --2- 172.21.15.107:0/459758239 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9cd4072110 0x7f9cbc003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:46.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd1d9b640 1 -- 172.21.15.107:0/459758239 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9cbc004d10 msgr2=0x7f9cbc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:46.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd1d9b640 1 --2- 172.21.15.107:0/459758239 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9cbc004d10 0x7f9cbc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:46.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.519+0000 7f9cd1d9b640 1 -- 172.21.15.107:0/459758239 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cbc003530 con 0x7f9cbc003fb0 2023-12-04T05:45:46.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd1d9b640 1 --2- 172.21.15.107:0/459758239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cbc003fb0 0x7f9cbc0047d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f9cc800ba00 tx=0x7f9cc801c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d672c9f9ba8cbcba server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:46.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd159a640 1 -- 172.21.15.107:0/459758239 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc80a9660 con 0x7f9cbc003fb0 2023-12-04T05:45:46.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd159a640 1 -- 172.21.15.107:0/459758239 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cc80a97c0 con 0x7f9cbc003fb0 2023-12-04T05:45:46.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd159a640 1 -- 172.21.15.107:0/459758239 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc80a9b70 con 0x7f9cbc003fb0 2023-12-04T05:45:46.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd8dd1640 1 -- 172.21.15.107:0/459758239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cbc003fb0 msgr2=0x7f9cbc0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:46.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd8dd1640 1 --2- 172.21.15.107:0/459758239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cbc003fb0 0x7f9cbc0047d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f9cc800ba00 tx=0x7f9cc801c680 comp rx=0 tx=0).stop 2023-12-04T05:45:46.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd2d9d640 1 -- 172.21.15.107:0/459758239 reap_dead start 2023-12-04T05:45:46.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd8dd1640 1 -- 172.21.15.107:0/459758239 shutdown_connections 2023-12-04T05:45:46.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd8dd1640 1 -- 172.21.15.107:0/459758239 >> 172.21.15.107:0/459758239 conn(0x7f9cd40c7ff0 msgr2=0x7f9cd40c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:46.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd8dd1640 1 -- 172.21.15.107:0/459758239 shutdown_connections 2023-12-04T05:45:46.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd8dd1640 1 -- 172.21.15.107:0/459758239 wait complete. 2023-12-04T05:45:46.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd8dd1640 1 Processor -- start 2023-12-04T05:45:46.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd8dd1640 1 -- start start 2023-12-04T05:45:46.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd8dd1640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9cbc003fb0 0x7f9cd41edb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:46.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd8dd1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cd4154db0 0x7f9cd4150260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:46.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd8dd1640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9cd414b2a0 0x7f9cd415a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:46.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd8dd1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd4155520 con 0x7f9cd4154db0 2023-12-04T05:45:46.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd8dd1640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd4159220 con 0x7f9cd414b2a0 2023-12-04T05:45:46.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd8dd1640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd41ee0b0 con 0x7f9cbc003fb0 2023-12-04T05:45:46.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cd4154db0 0x7f9cd4150260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:46.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cd4154db0 0x7f9cd4150260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37248/0 (socket says 172.21.15.107:37248) 2023-12-04T05:45:46.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd1d9b640 1 -- 172.21.15.107:0/1670278762 learned_addr learned my addr 172.21.15.107:0/1670278762 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:46.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd259c640 1 --2- 172.21.15.107:0/1670278762 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9cbc003fb0 0x7f9cd41edb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:46.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd2d9d640 1 --2- 172.21.15.107:0/1670278762 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9cd414b2a0 0x7f9cd415a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:46.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd1d9b640 1 -- 172.21.15.107:0/1670278762 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9cbc003fb0 msgr2=0x7f9cd41edb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:46.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd1d9b640 1 --2- 172.21.15.107:0/1670278762 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9cbc003fb0 0x7f9cd41edb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:46.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd1d9b640 1 -- 172.21.15.107:0/1670278762 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9cd414b2a0 msgr2=0x7f9cd415a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:46.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd1d9b640 1 --2- 172.21.15.107:0/1670278762 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9cd414b2a0 0x7f9cd415a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:46.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd1d9b640 1 -- 172.21.15.107:0/1670278762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd41f1150 con 0x7f9cd4154db0 2023-12-04T05:45:46.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.523+0000 7f9cd1d9b640 1 --2- 172.21.15.107:0/1670278762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cd4154db0 0x7f9cd4150260 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f9cc801cb60 tx=0x7f9cbc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:46.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.527+0000 7f9cc2ffd640 1 -- 172.21.15.107:0/1670278762 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc80b3860 con 0x7f9cd4154db0 2023-12-04T05:45:46.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.527+0000 7f9cc2ffd640 1 -- 172.21.15.107:0/1670278762 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cc80b39c0 con 0x7f9cd4154db0 2023-12-04T05:45:46.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.527+0000 7f9cd8dd1640 1 -- 172.21.15.107:0/1670278762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cd41f15e0 con 0x7f9cd4154db0 2023-12-04T05:45:46.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.527+0000 7f9cc2ffd640 1 -- 172.21.15.107:0/1670278762 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc80b3d50 con 0x7f9cd4154db0 2023-12-04T05:45:46.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.527+0000 7f9cd8dd1640 1 -- 172.21.15.107:0/1670278762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cd41f0820 con 0x7f9cd4154db0 2023-12-04T05:45:46.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.527+0000 7f9cd8dd1640 1 -- 172.21.15.107:0/1670278762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cd41f12e0 con 0x7f9cd4154db0 2023-12-04T05:45:46.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.527+0000 7f9cc2ffd640 1 -- 172.21.15.107:0/1670278762 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9cc80aa3f0 con 0x7f9cd4154db0 2023-12-04T05:45:46.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.527+0000 7f9cc2ffd640 1 --2- 172.21.15.107:0/1670278762 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9cb4066900 0x7f9cb4068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:46.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.527+0000 7f9cc2ffd640 1 -- 172.21.15.107:0/1670278762 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f9cc811f290 con 0x7f9cd4154db0 2023-12-04T05:45:46.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.527+0000 7f9cd259c640 1 --2- 172.21.15.107:0/1670278762 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9cb4066900 0x7f9cb4068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:46.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.527+0000 7f9cd259c640 1 --2- 172.21.15.107:0/1670278762 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9cb4066900 0x7f9cb4068dc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9cc400b8d0 tx=0x7f9cc4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:46.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.531+0000 7f9cc2ffd640 1 -- 172.21.15.107:0/1670278762 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9cc80ec2c0 con 0x7f9cd4154db0 2023-12-04T05:45:46.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.723+0000 7f9cd8dd1640 1 -- 172.21.15.107:0/1670278762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cd4067870 con 0x7f9cd4154db0 2023-12-04T05:45:46.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.723+0000 7f9cc2ffd640 1 -- 172.21.15.107:0/1670278762 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4959 (secure 0 0 0) 0x7f9cc80ef8e0 con 0x7f9cd4154db0 2023-12-04T05:45:46.727 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:46.727 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5190,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:38.916494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4288,"mds_1":5106,"mds_2":4263},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:stopping","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":36,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6839/3319682203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":3319682203},{"type":"v1","addr":"172.21.15.107:6839","nonce":3319682203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:46.727 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 44 2023-12-04T05:45:46.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.727+0000 7f9cd8dd1640 1 -- 172.21.15.107:0/1670278762 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9cb4066900 msgr2=0x7f9cb4068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:46.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.727+0000 7f9cd8dd1640 1 --2- 172.21.15.107:0/1670278762 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9cb4066900 0x7f9cb4068dc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9cc400b8d0 tx=0x7f9cc4008040 comp rx=0 tx=0).stop 2023-12-04T05:45:46.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.727+0000 7f9cd8dd1640 1 -- 172.21.15.107:0/1670278762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cd4154db0 msgr2=0x7f9cd4150260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:46.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.727+0000 7f9cd8dd1640 1 --2- 172.21.15.107:0/1670278762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cd4154db0 0x7f9cd4150260 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f9cc801cb60 tx=0x7f9cbc005800 comp rx=0 tx=0).stop 2023-12-04T05:45:46.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.727+0000 7f9cd8dd1640 1 -- 172.21.15.107:0/1670278762 shutdown_connections 2023-12-04T05:45:46.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.727+0000 7f9cd8dd1640 1 --2- 172.21.15.107:0/1670278762 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9cb4066900 0x7f9cb4068dc0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:46.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.727+0000 7f9cd8dd1640 1 --2- 172.21.15.107:0/1670278762 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9cd414b2a0 0x7f9cd415a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:46.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.727+0000 7f9cd8dd1640 1 --2- 172.21.15.107:0/1670278762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cd4154db0 0x7f9cd4150260 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:46.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.727+0000 7f9cd8dd1640 1 --2- 172.21.15.107:0/1670278762 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9cbc003fb0 0x7f9cd41edb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:46.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.727+0000 7f9cd8dd1640 1 -- 172.21.15.107:0/1670278762 >> 172.21.15.107:0/1670278762 conn(0x7f9cd40c7ff0 msgr2=0x7f9cd40c71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:46.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.727+0000 7f9cd8dd1640 1 -- 172.21.15.107:0/1670278762 shutdown_connections 2023-12-04T05:45:46.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:46.727+0000 7f9cd8dd1640 1 -- 172.21.15.107:0/1670278762 wait complete. 2023-12-04T05:45:46.747 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:45:48.357 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~7s 2023-12-04T05:45:48.357 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~121s 2023-12-04T05:45:48.357 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~162s 2023-12-04T05:45:48.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:48.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.871+0000 7f5b208d1640 1 Processor -- start 2023-12-04T05:45:48.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b208d1640 1 -- start start 2023-12-04T05:45:48.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b208d1640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5b1c0c4a80 0x7f5b1c0c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:48.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b208d1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1c06e660 con 0x7f5b1c157180 2023-12-04T05:45:48.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b208d1640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1c156cf0 con 0x7f5b1c0c46a0 2023-12-04T05:45:48.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b208d1640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c156490 con 0x7f5b1c0c4a80 2023-12-04T05:45:48.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b19d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5b1c157180 0x7f5b1c0c50e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50722/0 (socket says 172.21.15.107:50722) 2023-12-04T05:45:48.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b19d9b640 1 -- 172.21.15.107:0/3102012334 learned_addr learned my addr 172.21.15.107:0/3102012334 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:45:48.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b1959a640 1 -- 172.21.15.107:0/3102012334 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3466811857 0 0) 0x7f5b04001120 con 0x7f5b1c157180 2023-12-04T05:45:48.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b1959a640 1 -- 172.21.15.107:0/3102012334 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b08003530 con 0x7f5b1c157180 2023-12-04T05:45:48.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b1ad9d640 1 --2- 172.21.15.107:0/3102012334 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5b1c0c4a80 0x7f5b1c0c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:48.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b1959a640 1 -- 172.21.15.107:0/3102012334 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2133302728 0 0) 0x7f5b08003530 con 0x7f5b1c157180 2023-12-04T05:45:48.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b1959a640 1 -- 172.21.15.107:0/3102012334 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5b1c0c4a80 msgr2=0x7f5b1c0c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:48.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b1959a640 1 --2- 172.21.15.107:0/3102012334 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5b1c0c4a80 0x7f5b1c0c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:48.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b1959a640 1 -- 172.21.15.107:0/3102012334 >> v1:172.21.15.195:6789/0 conn(0x7f5b1c0c46a0 legacy=0x7f5b1c0c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:48.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b1959a640 1 -- 172.21.15.107:0/3102012334 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c06e0a0 con 0x7f5b1c157180 2023-12-04T05:45:48.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b1959a640 1 -- 172.21.15.107:0/3102012334 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f5b04001810 con 0x7f5b1c157180 2023-12-04T05:45:48.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b1959a640 1 -- 172.21.15.107:0/3102012334 >> v1:172.21.15.107:6789/0 conn(0x7f5b1c157180 legacy=0x7f5b1c0c50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:48.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b1959a640 1 --2- 172.21.15.107:0/3102012334 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b080039b0 0x7f5b08003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:48.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b1959a640 1 --2- 172.21.15.107:0/3102012334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b080047d0 0x7f5b08004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:48.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b1959a640 1 --2- 172.21.15.107:0/3102012334 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5b1c0c4a80 0x7f5b080051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:48.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b1959a640 1 -- 172.21.15.107:0/3102012334 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c156490 con 0x7f5b080047d0 2023-12-04T05:45:48.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b1959a640 1 -- 172.21.15.107:0/3102012334 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0800cbd0 con 0x7f5b080039b0 2023-12-04T05:45:48.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b1959a640 1 -- 172.21.15.107:0/3102012334 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b080059e0 con 0x7f5b1c0c4a80 2023-12-04T05:45:48.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b19d9b640 1 --2- 172.21.15.107:0/3102012334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b080047d0 0x7f5b08004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:48.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b1a59c640 1 --2- 172.21.15.107:0/3102012334 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b080039b0 0x7f5b08003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:48.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b1ad9d640 1 --2- 172.21.15.107:0/3102012334 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5b1c0c4a80 0x7f5b080051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:48.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b19d9b640 1 -- 172.21.15.107:0/3102012334 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5b1c0c4a80 msgr2=0x7f5b080051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:48.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b19d9b640 1 --2- 172.21.15.107:0/3102012334 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5b1c0c4a80 0x7f5b080051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:48.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b19d9b640 1 -- 172.21.15.107:0/3102012334 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b080039b0 msgr2=0x7f5b08003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:48.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b19d9b640 1 --2- 172.21.15.107:0/3102012334 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b080039b0 0x7f5b08003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:48.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.875+0000 7f5b19d9b640 1 -- 172.21.15.107:0/3102012334 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b08003530 con 0x7f5b080047d0 2023-12-04T05:45:48.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b19d9b640 1 --2- 172.21.15.107:0/3102012334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b080047d0 0x7f5b08004c50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5b0400c240 tx=0x7f5b0401f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c298a179dac331b7 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:48.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b1959a640 1 -- 172.21.15.107:0/3102012334 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b04005ae0 con 0x7f5b080047d0 2023-12-04T05:45:48.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b1959a640 1 -- 172.21.15.107:0/3102012334 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b040ab070 con 0x7f5b080047d0 2023-12-04T05:45:48.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b1959a640 1 -- 172.21.15.107:0/3102012334 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b04005ea0 con 0x7f5b080047d0 2023-12-04T05:45:48.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b208d1640 1 -- 172.21.15.107:0/3102012334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b080047d0 msgr2=0x7f5b08004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:48.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b208d1640 1 --2- 172.21.15.107:0/3102012334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b080047d0 0x7f5b08004c50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5b0400c240 tx=0x7f5b0401f3b0 comp rx=0 tx=0).stop 2023-12-04T05:45:48.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b1ad9d640 1 -- 172.21.15.107:0/3102012334 reap_dead start 2023-12-04T05:45:48.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b208d1640 1 -- 172.21.15.107:0/3102012334 shutdown_connections 2023-12-04T05:45:48.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b208d1640 1 -- 172.21.15.107:0/3102012334 >> 172.21.15.107:0/3102012334 conn(0x7f5b1c0c7ff0 msgr2=0x7f5b1c0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:48.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b208d1640 1 -- 172.21.15.107:0/3102012334 shutdown_connections 2023-12-04T05:45:48.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b208d1640 1 -- 172.21.15.107:0/3102012334 wait complete. 2023-12-04T05:45:48.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b208d1640 1 Processor -- start 2023-12-04T05:45:48.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b208d1640 1 -- start start 2023-12-04T05:45:48.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b208d1640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5b080047d0 0x7f5b1c1eb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:48.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b208d1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b1c1ee710 0x7f5b1c14dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:48.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b208d1640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b1c1e0a90 0x7f5b1c1e0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:48.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b208d1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c155b20 con 0x7f5b1c1ee710 2023-12-04T05:45:48.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b208d1640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c14d090 con 0x7f5b1c1e0a90 2023-12-04T05:45:48.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b208d1640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c14d650 con 0x7f5b080047d0 2023-12-04T05:45:48.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b19d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b1c1ee710 0x7f5b1c14dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:48.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b19d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b1c1ee710 0x7f5b1c14dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55216/0 (socket says 172.21.15.107:55216) 2023-12-04T05:45:48.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b19d9b640 1 -- 172.21.15.107:0/134466273 learned_addr learned my addr 172.21.15.107:0/134466273 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:48.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b1a59c640 1 --2- 172.21.15.107:0/134466273 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5b080047d0 0x7f5b1c1eb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:48.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b1ad9d640 1 --2- 172.21.15.107:0/134466273 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b1c1e0a90 0x7f5b1c1e0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:48.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b19d9b640 1 -- 172.21.15.107:0/134466273 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5b080047d0 msgr2=0x7f5b1c1eb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:48.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b19d9b640 1 --2- 172.21.15.107:0/134466273 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5b080047d0 0x7f5b1c1eb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:48.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b19d9b640 1 -- 172.21.15.107:0/134466273 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b1c1e0a90 msgr2=0x7f5b1c1e0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:48.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b19d9b640 1 --2- 172.21.15.107:0/134466273 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b1c1e0a90 0x7f5b1c1e0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:48.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b19d9b640 1 -- 172.21.15.107:0/134466273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c1eb9e0 con 0x7f5b1c1ee710 2023-12-04T05:45:48.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b1a59c640 1 --2- 172.21.15.107:0/134466273 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5b080047d0 0x7f5b1c1eb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:45:48.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b19d9b640 1 --2- 172.21.15.107:0/134466273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b1c1ee710 0x7f5b1c14dae0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5b0400c240 tx=0x7f5b08005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:48.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.879+0000 7f5b12ffd640 1 -- 172.21.15.107:0/134466273 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b040015d0 con 0x7f5b1c1ee710 2023-12-04T05:45:48.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.883+0000 7f5b12ffd640 1 -- 172.21.15.107:0/134466273 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b04001730 con 0x7f5b1c1ee710 2023-12-04T05:45:48.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.883+0000 7f5b208d1640 1 -- 172.21.15.107:0/134466273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1c1ebcc0 con 0x7f5b1c1ee710 2023-12-04T05:45:48.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.883+0000 7f5b12ffd640 1 -- 172.21.15.107:0/134466273 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b04007270 con 0x7f5b1c1ee710 2023-12-04T05:45:48.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.883+0000 7f5b208d1640 1 -- 172.21.15.107:0/134466273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1c14d2e0 con 0x7f5b1c1ee710 2023-12-04T05:45:48.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.883+0000 7f5b208d1640 1 -- 172.21.15.107:0/134466273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1c14d4a0 con 0x7f5b1c1ee710 2023-12-04T05:45:48.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.883+0000 7f5b12ffd640 1 -- 172.21.15.107:0/134466273 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f5b040073d0 con 0x7f5b1c1ee710 2023-12-04T05:45:48.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.883+0000 7f5b12ffd640 1 --2- 172.21.15.107:0/134466273 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5afc0668d0 0x7f5afc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:48.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.883+0000 7f5b12ffd640 1 -- 172.21.15.107:0/134466273 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f5b040b7040 con 0x7f5b1c1ee710 2023-12-04T05:45:48.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.883+0000 7f5b1a59c640 1 --2- 172.21.15.107:0/134466273 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5afc0668d0 0x7f5afc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:48.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.883+0000 7f5b1a59c640 1 --2- 172.21.15.107:0/134466273 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5afc0668d0 0x7f5afc068d90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c001d90 tx=0x7f5b0c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:48.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:48.887+0000 7f5b12ffd640 1 -- 172.21.15.107:0/134466273 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5b040eea00 con 0x7f5b1c1ee710 2023-12-04T05:45:49.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:49.079+0000 7f5b208d1640 1 -- 172.21.15.107:0/134466273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b1c06e660 con 0x7f5b1c1ee710 2023-12-04T05:45:49.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:49.079+0000 7f5b12ffd640 1 -- 172.21.15.107:0/134466273 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4959 (secure 0 0 0) 0x7f5b1c06e660 con 0x7f5b1c1ee710 2023-12-04T05:45:49.083 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:49.084 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5190,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:38.916494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4288,"mds_1":5106,"mds_2":4263},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:stopping","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":36,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6839/3319682203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":3319682203},{"type":"v1","addr":"172.21.15.107:6839","nonce":3319682203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:49.085 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 44 2023-12-04T05:45:49.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:49.079+0000 7f5b208d1640 1 -- 172.21.15.107:0/134466273 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5afc0668d0 msgr2=0x7f5afc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:49.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:49.079+0000 7f5b208d1640 1 --2- 172.21.15.107:0/134466273 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5afc0668d0 0x7f5afc068d90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c001d90 tx=0x7f5b0c009040 comp rx=0 tx=0).stop 2023-12-04T05:45:49.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:49.079+0000 7f5b208d1640 1 -- 172.21.15.107:0/134466273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b1c1ee710 msgr2=0x7f5b1c14dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:49.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:49.079+0000 7f5b208d1640 1 --2- 172.21.15.107:0/134466273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b1c1ee710 0x7f5b1c14dae0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5b0400c240 tx=0x7f5b08005800 comp rx=0 tx=0).stop 2023-12-04T05:45:49.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:49.079+0000 7f5b208d1640 1 -- 172.21.15.107:0/134466273 shutdown_connections 2023-12-04T05:45:49.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:49.079+0000 7f5b208d1640 1 --2- 172.21.15.107:0/134466273 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5afc0668d0 0x7f5afc068d90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:49.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:49.079+0000 7f5b208d1640 1 --2- 172.21.15.107:0/134466273 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b1c1e0a90 0x7f5b1c1e0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:49.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:49.079+0000 7f5b208d1640 1 --2- 172.21.15.107:0/134466273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b1c1ee710 0x7f5b1c14dae0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:49.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:49.079+0000 7f5b208d1640 1 --2- 172.21.15.107:0/134466273 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5b080047d0 0x7f5b1c1eb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:49.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:49.079+0000 7f5b208d1640 1 -- 172.21.15.107:0/134466273 >> 172.21.15.107:0/134466273 conn(0x7f5b1c0c7ff0 msgr2=0x7f5b1c0c7460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:49.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:49.083+0000 7f5b208d1640 1 -- 172.21.15.107:0/134466273 shutdown_connections 2023-12-04T05:45:49.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:49.083+0000 7f5b208d1640 1 -- 172.21.15.107:0/134466273 wait complete. 2023-12-04T05:45:49.100 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:45:51.101 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:51.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.231+0000 7f015b1d9640 1 Processor -- start 2023-12-04T05:45:51.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.231+0000 7f015b1d9640 1 -- start start 2023-12-04T05:45:51.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.231+0000 7f015b1d9640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f0154071c00 0x7f01540c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:51.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.231+0000 7f015b1d9640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0154070380 0x7f01540707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:51.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.231+0000 7f015b1d9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01540700c0 con 0x7f0154151180 2023-12-04T05:45:51.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.231+0000 7f015b1d9640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0154156df0 con 0x7f0154071c00 2023-12-04T05:45:51.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.231+0000 7f015b1d9640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01541570d0 con 0x7f0154070380 2023-12-04T05:45:51.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.231+0000 7f0158f75640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0154151180 0x7f01540714f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50728/0 (socket says 172.21.15.107:50728) 2023-12-04T05:45:51.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.231+0000 7f0158f75640 1 -- 172.21.15.107:0/1116494609 learned_addr learned my addr 172.21.15.107:0/1116494609 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:45:51.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.231+0000 7f01537fe640 1 -- 172.21.15.107:0/1116494609 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982815713 0 0) 0x7f01540700c0 con 0x7f0154151180 2023-12-04T05:45:51.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.231+0000 7f01537fe640 1 -- 172.21.15.107:0/1116494609 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f013c003530 con 0x7f0154151180 2023-12-04T05:45:51.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.231+0000 7f0153fff640 1 --2- 172.21.15.107:0/1116494609 >> v2:172.21.15.195:3300/0 conn(0x7f0154071c00 0x7f01540c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:51.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f0159776640 1 --2- 172.21.15.107:0/1116494609 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0154070380 0x7f01540707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:51.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f01537fe640 1 -- 172.21.15.107:0/1116494609 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3741295992 0 0) 0x7f013c003530 con 0x7f0154151180 2023-12-04T05:45:51.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f01537fe640 1 -- 172.21.15.107:0/1116494609 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0154070380 msgr2=0x7f01540707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:51.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f01537fe640 1 --2- 172.21.15.107:0/1116494609 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0154070380 0x7f01540707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:51.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f01537fe640 1 -- 172.21.15.107:0/1116494609 >> v2:172.21.15.195:3300/0 conn(0x7f0154071c00 msgr2=0x7f01540c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:51.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f01537fe640 1 --2- 172.21.15.107:0/1116494609 >> v2:172.21.15.195:3300/0 conn(0x7f0154071c00 0x7f01540c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:51.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f01537fe640 1 -- 172.21.15.107:0/1116494609 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01541573b0 con 0x7f0154151180 2023-12-04T05:45:51.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f01537fe640 1 -- 172.21.15.107:0/1116494609 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f01440016a0 con 0x7f0154151180 2023-12-04T05:45:51.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f0153fff640 1 --2- 172.21.15.107:0/1116494609 >> v2:172.21.15.195:3300/0 conn(0x7f0154071c00 0x7f01540c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:45:51.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f01537fe640 1 -- 172.21.15.107:0/1116494609 >> v1:172.21.15.107:6789/0 conn(0x7f0154151180 legacy=0x7f01540714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:51.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f01537fe640 1 --2- 172.21.15.107:0/1116494609 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0154070380 0x7f013c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:51.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f01537fe640 1 --2- 172.21.15.107:0/1116494609 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f013c003fe0 0x7f013c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:51.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f01537fe640 1 --2- 172.21.15.107:0/1116494609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f013c004d10 0x7f013c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:51.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f01537fe640 1 -- 172.21.15.107:0/1116494609 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0154156df0 con 0x7f013c004d10 2023-12-04T05:45:51.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f01537fe640 1 -- 172.21.15.107:0/1116494609 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01541570d0 con 0x7f013c003fe0 2023-12-04T05:45:51.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f01537fe640 1 -- 172.21.15.107:0/1116494609 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f013c00bb50 con 0x7f0154070380 2023-12-04T05:45:51.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f0159776640 1 --2- 172.21.15.107:0/1116494609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f013c004d10 0x7f013c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:51.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f0158f75640 1 --2- 172.21.15.107:0/1116494609 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0154070380 0x7f013c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:51.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f0153fff640 1 --2- 172.21.15.107:0/1116494609 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f013c003fe0 0x7f013c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:51.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f0159776640 1 -- 172.21.15.107:0/1116494609 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0154070380 msgr2=0x7f013c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:51.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f0159776640 1 --2- 172.21.15.107:0/1116494609 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0154070380 0x7f013c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:51.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f0159776640 1 -- 172.21.15.107:0/1116494609 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f013c003fe0 msgr2=0x7f013c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:51.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f0159776640 1 --2- 172.21.15.107:0/1116494609 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f013c003fe0 0x7f013c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:51.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f0159776640 1 -- 172.21.15.107:0/1116494609 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f013c003530 con 0x7f013c004d10 2023-12-04T05:45:51.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f0153fff640 1 --2- 172.21.15.107:0/1116494609 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f013c003fe0 0x7f013c0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:45:51.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f0158f75640 1 --2- 172.21.15.107:0/1116494609 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0154070380 0x7f013c0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:45:51.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f0159776640 1 --2- 172.21.15.107:0/1116494609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f013c004d10 0x7f013c005190 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f014c007360 tx=0x7f014c01ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9590699ba1cb7e68 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:51.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f01537fe640 1 -- 172.21.15.107:0/1116494609 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c0a5d20 con 0x7f013c004d10 2023-12-04T05:45:51.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f01537fe640 1 -- 172.21.15.107:0/1116494609 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014c0a5e80 con 0x7f013c004d10 2023-12-04T05:45:51.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f01537fe640 1 -- 172.21.15.107:0/1116494609 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c0ad660 con 0x7f013c004d10 2023-12-04T05:45:51.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f015b1d9640 1 -- 172.21.15.107:0/1116494609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f013c004d10 msgr2=0x7f013c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:51.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f015b1d9640 1 --2- 172.21.15.107:0/1116494609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f013c004d10 0x7f013c005190 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f014c007360 tx=0x7f014c01ce20 comp rx=0 tx=0).stop 2023-12-04T05:45:51.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f0159776640 1 -- 172.21.15.107:0/1116494609 reap_dead start 2023-12-04T05:45:51.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f015b1d9640 1 -- 172.21.15.107:0/1116494609 shutdown_connections 2023-12-04T05:45:51.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f015b1d9640 1 -- 172.21.15.107:0/1116494609 >> 172.21.15.107:0/1116494609 conn(0x7f01540c7ff0 msgr2=0x7f01540c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:51.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f015b1d9640 1 -- 172.21.15.107:0/1116494609 shutdown_connections 2023-12-04T05:45:51.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.235+0000 7f015b1d9640 1 -- 172.21.15.107:0/1116494609 wait complete. 2023-12-04T05:45:51.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f015b1d9640 1 Processor -- start 2023-12-04T05:45:51.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f015b1d9640 1 -- start start 2023-12-04T05:45:51.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f015b1d9640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f013c004d10 0x7f015413ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f015b1d9640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f015413b260 0x7f01540c4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f015b1d9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01540c45c0 0x7f01541f10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f015b1d9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01540c56f0 con 0x7f01540c45c0 2023-12-04T05:45:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f015b1d9640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015413b6a0 con 0x7f015413b260 2023-12-04T05:45:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f015b1d9640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0154153ee0 con 0x7f013c004d10 2023-12-04T05:45:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f0159776640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01540c45c0 0x7f01541f10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f0159776640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01540c45c0 0x7f01541f10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55230/0 (socket says 172.21.15.107:55230) 2023-12-04T05:45:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f0159776640 1 -- 172.21.15.107:0/2017761585 learned_addr learned my addr 172.21.15.107:0/2017761585 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f0158f75640 1 --2- 172.21.15.107:0/2017761585 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f013c004d10 0x7f015413ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f0153fff640 1 --2- 172.21.15.107:0/2017761585 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f015413b260 0x7f01540c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f0159776640 1 -- 172.21.15.107:0/2017761585 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f013c004d10 msgr2=0x7f015413ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f0159776640 1 --2- 172.21.15.107:0/2017761585 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f013c004d10 0x7f015413ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f0159776640 1 -- 172.21.15.107:0/2017761585 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f015413b260 msgr2=0x7f01540c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f0159776640 1 --2- 172.21.15.107:0/2017761585 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f015413b260 0x7f01540c4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:51.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f0159776640 1 -- 172.21.15.107:0/2017761585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01540c4a70 con 0x7f01540c45c0 2023-12-04T05:45:51.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f0153fff640 1 --2- 172.21.15.107:0/2017761585 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f015413b260 0x7f01540c4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:45:51.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f0159776640 1 --2- 172.21.15.107:0/2017761585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01540c45c0 0x7f01541f10b0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f014c0b0040 tx=0x7f013c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:51.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f01517fa640 1 -- 172.21.15.107:0/2017761585 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c0add50 con 0x7f01540c45c0 2023-12-04T05:45:51.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f01517fa640 1 -- 172.21.15.107:0/2017761585 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014c0039a0 con 0x7f01540c45c0 2023-12-04T05:45:51.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f015b1d9640 1 -- 172.21.15.107:0/2017761585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0154147a20 con 0x7f01540c45c0 2023-12-04T05:45:51.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f01517fa640 1 -- 172.21.15.107:0/2017761585 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c003d50 con 0x7f01540c45c0 2023-12-04T05:45:51.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f015b1d9640 1 -- 172.21.15.107:0/2017761585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0154153920 con 0x7f01540c45c0 2023-12-04T05:45:51.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.239+0000 7f015b1d9640 1 -- 172.21.15.107:0/2017761585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0154067870 con 0x7f01540c45c0 2023-12-04T05:45:51.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.243+0000 7f01517fa640 1 -- 172.21.15.107:0/2017761585 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f014c0b4040 con 0x7f01540c45c0 2023-12-04T05:45:51.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.243+0000 7f01517fa640 1 --2- 172.21.15.107:0/2017761585 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0140066900 0x7f0140068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:51.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.243+0000 7f01517fa640 1 -- 172.21.15.107:0/2017761585 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f014c11e550 con 0x7f01540c45c0 2023-12-04T05:45:51.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.243+0000 7f0158f75640 1 --2- 172.21.15.107:0/2017761585 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0140066900 0x7f0140068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:51.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.243+0000 7f0158f75640 1 --2- 172.21.15.107:0/2017761585 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0140066900 0x7f0140068dc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0144000f70 tx=0x7f0144009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:51.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.247+0000 7f01517fa640 1 -- 172.21.15.107:0/2017761585 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f014c0eb610 con 0x7f01540c45c0 2023-12-04T05:45:51.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.427+0000 7f015b1d9640 1 -- 172.21.15.107:0/2017761585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01540c26d0 con 0x7f01540c45c0 2023-12-04T05:45:51.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.427+0000 7f01517fa640 1 -- 172.21.15.107:0/2017761585 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4959 (secure 0 0 0) 0x7f014c0eec30 con 0x7f01540c45c0 2023-12-04T05:45:51.429 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:51.429 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5190,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:38.916494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4288,"mds_1":5106,"mds_2":4263},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:stopping","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":36,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6839/3319682203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":3319682203},{"type":"v1","addr":"172.21.15.107:6839","nonce":3319682203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:51.429 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 44 2023-12-04T05:45:51.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.427+0000 7f015b1d9640 1 -- 172.21.15.107:0/2017761585 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0140066900 msgr2=0x7f0140068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:51.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.427+0000 7f015b1d9640 1 --2- 172.21.15.107:0/2017761585 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0140066900 0x7f0140068dc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0144000f70 tx=0x7f0144009040 comp rx=0 tx=0).stop 2023-12-04T05:45:51.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.427+0000 7f015b1d9640 1 -- 172.21.15.107:0/2017761585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01540c45c0 msgr2=0x7f01541f10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:51.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.427+0000 7f015b1d9640 1 --2- 172.21.15.107:0/2017761585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01540c45c0 0x7f01541f10b0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f014c0b0040 tx=0x7f013c005800 comp rx=0 tx=0).stop 2023-12-04T05:45:51.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.427+0000 7f015b1d9640 1 -- 172.21.15.107:0/2017761585 shutdown_connections 2023-12-04T05:45:51.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.427+0000 7f015b1d9640 1 --2- 172.21.15.107:0/2017761585 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0140066900 0x7f0140068dc0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:51.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.427+0000 7f015b1d9640 1 --2- 172.21.15.107:0/2017761585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01540c45c0 0x7f01541f10b0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:51.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.427+0000 7f015b1d9640 1 --2- 172.21.15.107:0/2017761585 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f015413b260 0x7f01540c4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:51.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.427+0000 7f015b1d9640 1 --2- 172.21.15.107:0/2017761585 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f013c004d10 0x7f015413ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:51.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.427+0000 7f015b1d9640 1 -- 172.21.15.107:0/2017761585 >> 172.21.15.107:0/2017761585 conn(0x7f01540c7ff0 msgr2=0x7f015406ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:51.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.427+0000 7f015b1d9640 1 -- 172.21.15.107:0/2017761585 shutdown_connections 2023-12-04T05:45:51.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:51.427+0000 7f015b1d9640 1 -- 172.21.15.107:0/2017761585 wait complete. 2023-12-04T05:45:51.446 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:45:53.447 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:53.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.574+0000 7f4f2c3e9640 1 Processor -- start 2023-12-04T05:45:53.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.574+0000 7f4f2c3e9640 1 -- start start 2023-12-04T05:45:53.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f2c3e9640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4f24072110 0x7f4f24158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:53.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f2c3e9640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4f240c56f0 0x7f4f24140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:53.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f2c3e9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f24158da0 con 0x7f4f240724f0 2023-12-04T05:45:53.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f2c3e9640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f241583d0 con 0x7f4f240c56f0 2023-12-04T05:45:53.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f2c3e9640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f240c6170 con 0x7f4f24072110 2023-12-04T05:45:53.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f29984640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4f240724f0 0x7f4f240c7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50744/0 (socket says 172.21.15.107:50744) 2023-12-04T05:45:53.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f29984640 1 -- 172.21.15.107:0/3749055729 learned_addr learned my addr 172.21.15.107:0/3749055729 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:45:53.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f2a185640 1 --2- 172.21.15.107:0/3749055729 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4f24072110 0x7f4f24158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:53.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f2a986640 1 --2- 172.21.15.107:0/3749055729 >> v2:172.21.15.195:3300/0 conn(0x7f4f240c56f0 0x7f4f24140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:53.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f29183640 1 -- 172.21.15.107:0/3749055729 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740850477 0 0) 0x7f4f14001120 con 0x7f4f240724f0 2023-12-04T05:45:53.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f29183640 1 -- 172.21.15.107:0/3749055729 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f10003530 con 0x7f4f240724f0 2023-12-04T05:45:53.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f29183640 1 -- 172.21.15.107:0/3749055729 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4127443050 0 0) 0x7f4f10003530 con 0x7f4f240724f0 2023-12-04T05:45:53.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f29183640 1 -- 172.21.15.107:0/3749055729 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4f24072110 msgr2=0x7f4f24158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:53.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f29183640 1 --2- 172.21.15.107:0/3749055729 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4f24072110 0x7f4f24158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:53.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f29183640 1 -- 172.21.15.107:0/3749055729 >> v2:172.21.15.195:3300/0 conn(0x7f4f240c56f0 msgr2=0x7f4f24140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:53.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f29183640 1 --2- 172.21.15.107:0/3749055729 >> v2:172.21.15.195:3300/0 conn(0x7f4f240c56f0 0x7f4f24140830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:53.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f29183640 1 -- 172.21.15.107:0/3749055729 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f24157da0 con 0x7f4f240724f0 2023-12-04T05:45:53.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f29183640 1 -- 172.21.15.107:0/3749055729 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f4f14001810 con 0x7f4f240724f0 2023-12-04T05:45:53.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f29183640 1 -- 172.21.15.107:0/3749055729 >> v1:172.21.15.107:6789/0 conn(0x7f4f240724f0 legacy=0x7f4f240c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:53.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f29183640 1 --2- 172.21.15.107:0/3749055729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f10003980 0x7f4f10003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:53.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f29183640 1 --2- 172.21.15.107:0/3749055729 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4f24072110 0x7f4f100047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:53.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f29183640 1 --2- 172.21.15.107:0/3749055729 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f10004d10 0x7f4f10005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:53.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f29183640 1 -- 172.21.15.107:0/3749055729 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f241583d0 con 0x7f4f10003980 2023-12-04T05:45:53.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f29183640 1 -- 172.21.15.107:0/3749055729 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f240c6170 con 0x7f4f10004d10 2023-12-04T05:45:53.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f29183640 1 -- 172.21.15.107:0/3749055729 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1000b990 con 0x7f4f24072110 2023-12-04T05:45:53.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f2a185640 1 --2- 172.21.15.107:0/3749055729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f10003980 0x7f4f10003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:53.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f29984640 1 --2- 172.21.15.107:0/3749055729 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4f24072110 0x7f4f100047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:53.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f2a185640 1 -- 172.21.15.107:0/3749055729 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4f24072110 msgr2=0x7f4f100047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:53.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f2a185640 1 --2- 172.21.15.107:0/3749055729 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4f24072110 0x7f4f100047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:53.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f2a986640 1 --2- 172.21.15.107:0/3749055729 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f10004d10 0x7f4f10005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:53.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f2a185640 1 -- 172.21.15.107:0/3749055729 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f10004d10 msgr2=0x7f4f10005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:53.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f2a185640 1 --2- 172.21.15.107:0/3749055729 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f10004d10 0x7f4f10005190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:53.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f2a185640 1 -- 172.21.15.107:0/3749055729 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f10003530 con 0x7f4f10003980 2023-12-04T05:45:53.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f29984640 1 --2- 172.21.15.107:0/3749055729 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4f24072110 0x7f4f100047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:45:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.578+0000 7f4f2a185640 1 --2- 172.21.15.107:0/3749055729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f10003980 0x7f4f10003da0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4f20009ae0 tx=0x7f4f2001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55f983d58897bb00 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f29183640 1 -- 172.21.15.107:0/3749055729 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f200a5510 con 0x7f4f10003980 2023-12-04T05:45:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f29183640 1 -- 172.21.15.107:0/3749055729 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f200a5670 con 0x7f4f10003980 2023-12-04T05:45:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f29183640 1 -- 172.21.15.107:0/3749055729 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f200a5a20 con 0x7f4f10003980 2023-12-04T05:45:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f2c3e9640 1 -- 172.21.15.107:0/3749055729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f10003980 msgr2=0x7f4f10003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f2c3e9640 1 --2- 172.21.15.107:0/3749055729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f10003980 0x7f4f10003da0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4f20009ae0 tx=0x7f4f2001ca50 comp rx=0 tx=0).stop 2023-12-04T05:45:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f2a986640 1 -- 172.21.15.107:0/3749055729 reap_dead start 2023-12-04T05:45:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f2c3e9640 1 -- 172.21.15.107:0/3749055729 shutdown_connections 2023-12-04T05:45:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f2c3e9640 1 -- 172.21.15.107:0/3749055729 >> 172.21.15.107:0/3749055729 conn(0x7f4f240c7ff0 msgr2=0x7f4f240c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f2c3e9640 1 -- 172.21.15.107:0/3749055729 shutdown_connections 2023-12-04T05:45:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f2c3e9640 1 -- 172.21.15.107:0/3749055729 wait complete. 2023-12-04T05:45:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f2c3e9640 1 Processor -- start 2023-12-04T05:45:53.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f2c3e9640 1 -- start start 2023-12-04T05:45:53.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f2c3e9640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4f24140100 0x7f4f240b9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:53.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f2c3e9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f241f2110 0x7f4f2415c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f2c3e9640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f2406d860 0x7f4f241e4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f2c3e9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f240b97a0 con 0x7f4f241f2110 2023-12-04T05:45:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f2c3e9640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2415c210 con 0x7f4f2406d860 2023-12-04T05:45:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f2c3e9640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2415b340 con 0x7f4f24140100 2023-12-04T05:45:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f29984640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f241f2110 0x7f4f2415c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f29984640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f241f2110 0x7f4f2415c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55252/0 (socket says 172.21.15.107:55252) 2023-12-04T05:45:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f29984640 1 -- 172.21.15.107:0/1608342681 learned_addr learned my addr 172.21.15.107:0/1608342681 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f2a185640 1 --2- 172.21.15.107:0/1608342681 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4f24140100 0x7f4f240b9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f2a986640 1 --2- 172.21.15.107:0/1608342681 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f2406d860 0x7f4f241e4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f29984640 1 -- 172.21.15.107:0/1608342681 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4f24140100 msgr2=0x7f4f240b9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f29984640 1 --2- 172.21.15.107:0/1608342681 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4f24140100 0x7f4f240b9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f29984640 1 -- 172.21.15.107:0/1608342681 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f2406d860 msgr2=0x7f4f241e4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f29984640 1 --2- 172.21.15.107:0/1608342681 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f2406d860 0x7f4f241e4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f29984640 1 -- 172.21.15.107:0/1608342681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2415b600 con 0x7f4f241f2110 2023-12-04T05:45:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f2a986640 1 --2- 172.21.15.107:0/1608342681 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f2406d860 0x7f4f241e4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:45:53.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f2a185640 1 --2- 172.21.15.107:0/1608342681 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4f24140100 0x7f4f240b9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:45:53.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f29984640 1 --2- 172.21.15.107:0/1608342681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f241f2110 0x7f4f2415c6a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4f14000f70 tx=0x7f4f1400df70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:53.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f1affd640 1 -- 172.21.15.107:0/1608342681 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f14004cc0 con 0x7f4f241f2110 2023-12-04T05:45:53.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f1affd640 1 -- 172.21.15.107:0/1608342681 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f14004e20 con 0x7f4f241f2110 2023-12-04T05:45:53.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f2c3e9640 1 -- 172.21.15.107:0/1608342681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f2415b8e0 con 0x7f4f241f2110 2023-12-04T05:45:53.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f1affd640 1 -- 172.21.15.107:0/1608342681 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f140072d0 con 0x7f4f241f2110 2023-12-04T05:45:53.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.582+0000 7f4f2c3e9640 1 -- 172.21.15.107:0/1608342681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f2415bea0 con 0x7f4f241f2110 2023-12-04T05:45:53.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.586+0000 7f4f2c3e9640 1 -- 172.21.15.107:0/1608342681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f2415c380 con 0x7f4f241f2110 2023-12-04T05:45:53.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.586+0000 7f4f1affd640 1 -- 172.21.15.107:0/1608342681 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4f14007430 con 0x7f4f241f2110 2023-12-04T05:45:53.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.586+0000 7f4f1affd640 1 --2- 172.21.15.107:0/1608342681 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4f08066900 0x7f4f08068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:53.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.586+0000 7f4f1affd640 1 -- 172.21.15.107:0/1608342681 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f4f14083d00 con 0x7f4f241f2110 2023-12-04T05:45:53.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.586+0000 7f4f2a185640 1 --2- 172.21.15.107:0/1608342681 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4f08066900 0x7f4f08068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:53.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.586+0000 7f4f2a185640 1 --2- 172.21.15.107:0/1608342681 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4f08066900 0x7f4f08068dc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4f200af000 tx=0x7f4f20008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:53.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.590+0000 7f4f1affd640 1 -- 172.21.15.107:0/1608342681 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4f14050dd0 con 0x7f4f241f2110 2023-12-04T05:45:53.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.778+0000 7f4f2c3e9640 1 -- 172.21.15.107:0/1608342681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f24158da0 con 0x7f4f241f2110 2023-12-04T05:45:53.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.778+0000 7f4f1affd640 1 -- 172.21.15.107:0/1608342681 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4959 (secure 0 0 0) 0x7f4f24158da0 con 0x7f4f241f2110 2023-12-04T05:45:53.782 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:53.783 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5190,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:38.916494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4288,"mds_1":5106,"mds_2":4263},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:stopping","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":36,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6839/3319682203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":3319682203},{"type":"v1","addr":"172.21.15.107:6839","nonce":3319682203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:53.783 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 44 2023-12-04T05:45:53.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.782+0000 7f4f2c3e9640 1 -- 172.21.15.107:0/1608342681 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4f08066900 msgr2=0x7f4f08068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:53.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.782+0000 7f4f2c3e9640 1 --2- 172.21.15.107:0/1608342681 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4f08066900 0x7f4f08068dc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4f200af000 tx=0x7f4f20008040 comp rx=0 tx=0).stop 2023-12-04T05:45:53.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.782+0000 7f4f2c3e9640 1 -- 172.21.15.107:0/1608342681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f241f2110 msgr2=0x7f4f2415c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:53.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.782+0000 7f4f2c3e9640 1 --2- 172.21.15.107:0/1608342681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f241f2110 0x7f4f2415c6a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4f14000f70 tx=0x7f4f1400df70 comp rx=0 tx=0).stop 2023-12-04T05:45:53.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.782+0000 7f4f2c3e9640 1 -- 172.21.15.107:0/1608342681 shutdown_connections 2023-12-04T05:45:53.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.782+0000 7f4f2c3e9640 1 --2- 172.21.15.107:0/1608342681 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4f08066900 0x7f4f08068dc0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:53.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.782+0000 7f4f2c3e9640 1 --2- 172.21.15.107:0/1608342681 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4f2406d860 0x7f4f241e4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:53.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.782+0000 7f4f2c3e9640 1 --2- 172.21.15.107:0/1608342681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f241f2110 0x7f4f2415c6a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:53.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.782+0000 7f4f2c3e9640 1 --2- 172.21.15.107:0/1608342681 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4f24140100 0x7f4f240b9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:53.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.782+0000 7f4f2c3e9640 1 -- 172.21.15.107:0/1608342681 >> 172.21.15.107:0/1608342681 conn(0x7f4f240c7ff0 msgr2=0x7f4f24156b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:53.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.782+0000 7f4f2c3e9640 1 -- 172.21.15.107:0/1608342681 shutdown_connections 2023-12-04T05:45:53.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:53.782+0000 7f4f2c3e9640 1 -- 172.21.15.107:0/1608342681 wait complete. 2023-12-04T05:45:53.801 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-12-04T05:45:53.801 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 44, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-12-04T05:39:59.032019+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 43, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4263": { "addr": "172.21.15.107:6837/2210527429", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6835", "nonce": 2210527429, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 2210527429, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4263, "incarnation": 12, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:stopping", "state_seq": 7 }, "gid_4288": { "addr": "172.21.15.195:6837/3412865477", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6835", "nonce": 3412865477, "type": "v2" }, { "addr": "172.21.15.195:6837", "nonce": 3412865477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_5106": { "addr": "172.21.15.107:6839/3319682203", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6838", "nonce": 3319682203, "type": "v2" }, { "addr": "172.21.15.107:6839", "nonce": 3319682203, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5106, "incarnation": 36, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 26, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-12-04T05:45:38.916494+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 3, 4 ], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 5106, "mds_2": 4263 } } } ], "standbys": [ { "addr": "172.21.15.107:6836/2928623624", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 2928623624, "type": "v2" }, { "addr": "172.21.15.107:6836", "nonce": 2928623624, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 44, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5190, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-12-04T05:45:54.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~13s 2023-12-04T05:45:54.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~127s 2023-12-04T05:45:54.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~168s 2023-12-04T05:45:55.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:55.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.934+0000 7fe9dc956640 1 Processor -- start 2023-12-04T05:45:55.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.934+0000 7fe9dc956640 1 -- start start 2023-12-04T05:45:55.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.934+0000 7fe9dc956640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe9d80d01c0 0x7fe9d8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:55.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.934+0000 7fe9dc956640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe9d8072220 0x7fe9d80c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:55.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.934+0000 7fe9dc956640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d815d710 con 0x7fe9d8154020 2023-12-04T05:45:55.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.934+0000 7fe9dc956640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d80ca230 con 0x7fe9d80d01c0 2023-12-04T05:45:55.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.934+0000 7fe9dc956640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d80c6770 con 0x7fe9d8072220 2023-12-04T05:45:55.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.934+0000 7fe9d5d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe9d80d01c0 0x7fe9d8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:55.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.934+0000 7fe9d659c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe9d8154020 0x7fe9d815a810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50756/0 (socket says 172.21.15.107:50756) 2023-12-04T05:45:55.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.934+0000 7fe9d5d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe9d80d01c0 0x7fe9d8071ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55260/0 (socket says 172.21.15.107:55260) 2023-12-04T05:45:55.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.934+0000 7fe9d659c640 1 -- 172.21.15.107:0/224998243 learned_addr learned my addr 172.21.15.107:0/224998243 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:45:55.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.934+0000 7fe9d6d9d640 1 --2- 172.21.15.107:0/224998243 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe9d8072220 0x7fe9d80c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:55.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.934+0000 7fe9d5d9b640 1 -- 172.21.15.107:0/224998243 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe9d8072220 msgr2=0x7fe9d80c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:55.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.934+0000 7fe9d5d9b640 1 --2- 172.21.15.107:0/224998243 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe9d8072220 0x7fe9d80c4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:55.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.934+0000 7fe9d5d9b640 1 -- 172.21.15.107:0/224998243 >> v1:172.21.15.107:6789/0 conn(0x7fe9d8154020 legacy=0x7fe9d815a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:55.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.934+0000 7fe9d5d9b640 1 -- 172.21.15.107:0/224998243 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d815d9f0 con 0x7fe9d80d01c0 2023-12-04T05:45:55.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.934+0000 7fe9d6d9d640 1 --2- 172.21.15.107:0/224998243 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe9d8072220 0x7fe9d80c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:45:55.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.938+0000 7fe9d5d9b640 1 --2- 172.21.15.107:0/224998243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9d80d01c0 0x7fe9d8071ce0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc0097f0 tx=0x7fe9cc01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8988ed655311a6d0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:55.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.938+0000 7fe9d559a640 1 -- 172.21.15.107:0/224998243 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc0a8970 con 0x7fe9d80d01c0 2023-12-04T05:45:55.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.938+0000 7fe9d559a640 1 -- 172.21.15.107:0/224998243 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9cc0a8ad0 con 0x7fe9d80d01c0 2023-12-04T05:45:55.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.938+0000 7fe9d559a640 1 -- 172.21.15.107:0/224998243 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc0a8e60 con 0x7fe9d80d01c0 2023-12-04T05:45:55.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.938+0000 7fe9dc956640 1 -- 172.21.15.107:0/224998243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9d80d01c0 msgr2=0x7fe9d8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:55.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.938+0000 7fe9dc956640 1 --2- 172.21.15.107:0/224998243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9d80d01c0 0x7fe9d8071ce0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc0097f0 tx=0x7fe9cc01ec70 comp rx=0 tx=0).stop 2023-12-04T05:45:55.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.938+0000 7fe9dc956640 1 -- 172.21.15.107:0/224998243 shutdown_connections 2023-12-04T05:45:55.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.938+0000 7fe9dc956640 1 --2- 172.21.15.107:0/224998243 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe9d8072220 0x7fe9d80c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:55.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.938+0000 7fe9dc956640 1 --2- 172.21.15.107:0/224998243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9d80d01c0 0x7fe9d8071ce0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:55.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.938+0000 7fe9dc956640 1 -- 172.21.15.107:0/224998243 >> 172.21.15.107:0/224998243 conn(0x7fe9d806c6c0 msgr2=0x7fe9d80cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:55.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.938+0000 7fe9dc956640 1 -- 172.21.15.107:0/224998243 shutdown_connections 2023-12-04T05:45:55.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.938+0000 7fe9dc956640 1 -- 172.21.15.107:0/224998243 wait complete. 2023-12-04T05:45:55.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.938+0000 7fe9dc956640 1 Processor -- start 2023-12-04T05:45:55.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.938+0000 7fe9dc956640 1 -- start start 2023-12-04T05:45:55.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9dc956640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe9d8072220 0x7fe9d80ce840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:55.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9dc956640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe9d80d01c0 0x7fe9d8158360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:55.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9dc956640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9d8154020 0x7fe9d81563f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:55.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9dc956640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d8155fd0 con 0x7fe9d8154020 2023-12-04T05:45:55.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9dc956640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d815a4d0 con 0x7fe9d80d01c0 2023-12-04T05:45:55.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9dc956640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d81556c0 con 0x7fe9d8072220 2023-12-04T05:45:55.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9d6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9d8154020 0x7fe9d81563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:55.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9d6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9d8154020 0x7fe9d81563f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55266/0 (socket says 172.21.15.107:55266) 2023-12-04T05:45:55.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9d6d9d640 1 -- 172.21.15.107:0/1016571329 learned_addr learned my addr 172.21.15.107:0/1016571329 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:55.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9d659c640 1 --2- 172.21.15.107:0/1016571329 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe9d8072220 0x7fe9d80ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:55.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9d6d9d640 1 -- 172.21.15.107:0/1016571329 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe9d8072220 msgr2=0x7fe9d80ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:55.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9d6d9d640 1 --2- 172.21.15.107:0/1016571329 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe9d8072220 0x7fe9d80ce840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:55.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9d6d9d640 1 -- 172.21.15.107:0/1016571329 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe9d80d01c0 msgr2=0x7fe9d8158360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:55.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9d6d9d640 1 --2- 172.21.15.107:0/1016571329 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe9d80d01c0 0x7fe9d8158360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:55.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9d6d9d640 1 -- 172.21.15.107:0/1016571329 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9cc009420 con 0x7fe9d8154020 2023-12-04T05:45:55.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9d6d9d640 1 --2- 172.21.15.107:0/1016571329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9d8154020 0x7fe9d81563f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fe9d000cef0 tx=0x7fe9d000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:55.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9c6ffd640 1 -- 172.21.15.107:0/1016571329 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d000d840 con 0x7fe9d8154020 2023-12-04T05:45:55.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9c6ffd640 1 -- 172.21.15.107:0/1016571329 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d000d9c0 con 0x7fe9d8154020 2023-12-04T05:45:55.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9dc956640 1 -- 172.21.15.107:0/1016571329 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d8155980 con 0x7fe9d8154020 2023-12-04T05:45:55.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9c6ffd640 1 -- 172.21.15.107:0/1016571329 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d000dd70 con 0x7fe9d8154020 2023-12-04T05:45:55.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9dc956640 1 -- 172.21.15.107:0/1016571329 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d81590b0 con 0x7fe9d8154020 2023-12-04T05:45:55.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9dc956640 1 -- 172.21.15.107:0/1016571329 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d8156110 con 0x7fe9d8154020 2023-12-04T05:45:55.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9c6ffd640 1 -- 172.21.15.107:0/1016571329 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe9d001e3b0 con 0x7fe9d8154020 2023-12-04T05:45:55.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9c6ffd640 1 --2- 172.21.15.107:0/1016571329 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe9b80668d0 0x7fe9b8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:55.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9c6ffd640 1 -- 172.21.15.107:0/1016571329 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fe9d0085070 con 0x7fe9d8154020 2023-12-04T05:45:55.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9d659c640 1 --2- 172.21.15.107:0/1016571329 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe9b80668d0 0x7fe9b8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:55.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.942+0000 7fe9d659c640 1 --2- 172.21.15.107:0/1016571329 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe9b80668d0 0x7fe9b8068d90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8000f70 tx=0x7fe9c8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:55.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:55.946+0000 7fe9c6ffd640 1 -- 172.21.15.107:0/1016571329 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe9d8156110 con 0x7fe9d8154020 2023-12-04T05:45:56.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:56.134+0000 7fe9dc956640 1 -- 172.21.15.107:0/1016571329 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9d8156110 con 0x7fe9d8154020 2023-12-04T05:45:56.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:56.134+0000 7fe9c6ffd640 1 -- 172.21.15.107:0/1016571329 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4166 (secure 0 0 0) 0x7fe9d8156110 con 0x7fe9d8154020 2023-12-04T05:45:56.137 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:56.137 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:54.798865+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4288,"mds_1":5190,"mds_2":4263},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:stopping","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:56.137 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 45 2023-12-04T05:45:56.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:56.134+0000 7fe9dc956640 1 -- 172.21.15.107:0/1016571329 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe9b80668d0 msgr2=0x7fe9b8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:56.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:56.134+0000 7fe9dc956640 1 --2- 172.21.15.107:0/1016571329 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe9b80668d0 0x7fe9b8068d90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8000f70 tx=0x7fe9c8009040 comp rx=0 tx=0).stop 2023-12-04T05:45:56.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:56.134+0000 7fe9dc956640 1 -- 172.21.15.107:0/1016571329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9d8154020 msgr2=0x7fe9d81563f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:56.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:56.134+0000 7fe9dc956640 1 --2- 172.21.15.107:0/1016571329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9d8154020 0x7fe9d81563f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fe9d000cef0 tx=0x7fe9d000a710 comp rx=0 tx=0).stop 2023-12-04T05:45:56.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:56.134+0000 7fe9dc956640 1 -- 172.21.15.107:0/1016571329 shutdown_connections 2023-12-04T05:45:56.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:56.134+0000 7fe9dc956640 1 --2- 172.21.15.107:0/1016571329 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe9b80668d0 0x7fe9b8068d90 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:56.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:56.134+0000 7fe9dc956640 1 --2- 172.21.15.107:0/1016571329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9d8154020 0x7fe9d81563f0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:56.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:56.134+0000 7fe9dc956640 1 --2- 172.21.15.107:0/1016571329 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe9d80d01c0 0x7fe9d8158360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:56.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:56.134+0000 7fe9dc956640 1 --2- 172.21.15.107:0/1016571329 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe9d8072220 0x7fe9d80ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:56.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:56.134+0000 7fe9dc956640 1 -- 172.21.15.107:0/1016571329 >> 172.21.15.107:0/1016571329 conn(0x7fe9d806c6c0 msgr2=0x7fe9d81521e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:56.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:56.134+0000 7fe9dc956640 1 -- 172.21.15.107:0/1016571329 shutdown_connections 2023-12-04T05:45:56.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:56.138+0000 7fe9dc956640 1 -- 172.21.15.107:0/1016571329 wait complete. 2023-12-04T05:45:56.155 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:45:58.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:45:58.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.282+0000 7f5cabd7a640 1 Processor -- start 2023-12-04T05:45:58.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.282+0000 7f5cabd7a640 1 -- start start 2023-12-04T05:45:58.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.286+0000 7f5cabd7a640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5ca4151180 0x7f5ca40c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:58.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.286+0000 7f5cabd7a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5ca40714f0 0x7f5ca4071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:58.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.286+0000 7f5cabd7a640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5ca4070380 0x7f5ca40707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:58.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.286+0000 7f5cabd7a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ca40700c0 con 0x7f5ca40714f0 2023-12-04T05:45:58.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.286+0000 7f5cabd7a640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ca4071e60 con 0x7f5ca4151180 2023-12-04T05:45:58.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.286+0000 7f5cabd7a640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca4156df0 con 0x7f5ca4070380 2023-12-04T05:45:58.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.286+0000 7f5ca9315640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5ca40714f0 0x7f5ca4071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:58.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.286+0000 7f5ca9315640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5ca40714f0 0x7f5ca4071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41364/0 (socket says 172.21.15.107:41364) 2023-12-04T05:45:58.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.286+0000 7f5ca9315640 1 -- 172.21.15.107:0/1602224887 learned_addr learned my addr 172.21.15.107:0/1602224887 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:58.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.286+0000 7f5caa317640 1 --2- 172.21.15.107:0/1602224887 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5ca4070380 0x7f5ca40707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:58.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.286+0000 7f5ca9b16640 1 --2- 172.21.15.107:0/1602224887 >> v2:172.21.15.195:3300/0 conn(0x7f5ca4151180 0x7f5ca40c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:58.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.286+0000 7f5ca9315640 1 -- 172.21.15.107:0/1602224887 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5ca4070380 msgr2=0x7f5ca40707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:58.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.286+0000 7f5ca9315640 1 --2- 172.21.15.107:0/1602224887 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5ca4070380 0x7f5ca40707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:58.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.286+0000 7f5ca9315640 1 -- 172.21.15.107:0/1602224887 >> v2:172.21.15.195:3300/0 conn(0x7f5ca4151180 msgr2=0x7f5ca40c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:45:58.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.286+0000 7f5ca9315640 1 --2- 172.21.15.107:0/1602224887 >> v2:172.21.15.195:3300/0 conn(0x7f5ca4151180 0x7f5ca40c79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:58.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.286+0000 7f5ca9315640 1 -- 172.21.15.107:0/1602224887 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca41570d0 con 0x7f5ca40714f0 2023-12-04T05:45:58.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.286+0000 7f5ca9315640 1 --2- 172.21.15.107:0/1602224887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ca40714f0 0x7f5ca4071920 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5ca406ba20 tx=0x7f5c9401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fceeb8294568c56 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:58.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.286+0000 7f5ca8b14640 1 -- 172.21.15.107:0/1602224887 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c940a89b0 con 0x7f5ca40714f0 2023-12-04T05:45:58.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.286+0000 7f5ca8b14640 1 -- 172.21.15.107:0/1602224887 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c940a8b10 con 0x7f5ca40714f0 2023-12-04T05:45:58.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.286+0000 7f5ca8b14640 1 -- 172.21.15.107:0/1602224887 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c940a8ea0 con 0x7f5ca40714f0 2023-12-04T05:45:58.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5cabd7a640 1 -- 172.21.15.107:0/1602224887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ca40714f0 msgr2=0x7f5ca4071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:58.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5cabd7a640 1 --2- 172.21.15.107:0/1602224887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ca40714f0 0x7f5ca4071920 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5ca406ba20 tx=0x7f5c9401ecf0 comp rx=0 tx=0).stop 2023-12-04T05:45:58.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5cabd7a640 1 -- 172.21.15.107:0/1602224887 shutdown_connections 2023-12-04T05:45:58.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5cabd7a640 1 --2- 172.21.15.107:0/1602224887 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5ca4070380 0x7f5ca40707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:58.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5cabd7a640 1 --2- 172.21.15.107:0/1602224887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ca40714f0 0x7f5ca4071920 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:58.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5cabd7a640 1 --2- 172.21.15.107:0/1602224887 >> v2:172.21.15.195:3300/0 conn(0x7f5ca4151180 0x7f5ca40c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:58.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5cabd7a640 1 -- 172.21.15.107:0/1602224887 >> 172.21.15.107:0/1602224887 conn(0x7f5ca40c7ff0 msgr2=0x7f5ca40c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:58.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5cabd7a640 1 -- 172.21.15.107:0/1602224887 shutdown_connections 2023-12-04T05:45:58.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5cabd7a640 1 -- 172.21.15.107:0/1602224887 wait complete. 2023-12-04T05:45:58.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5cabd7a640 1 Processor -- start 2023-12-04T05:45:58.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5cabd7a640 1 -- start start 2023-12-04T05:45:58.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5cabd7a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ca4070380 0x7f5ca41f11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:58.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5cabd7a640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5ca40714f0 0x7f5ca41f1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:58.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5cabd7a640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ca4151180 0x7f5ca40c4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:58.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5cabd7a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca41f1da0 con 0x7f5ca4070380 2023-12-04T05:45:58.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5cabd7a640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca414b580 con 0x7f5ca4151180 2023-12-04T05:45:58.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5cabd7a640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca40c3c90 con 0x7f5ca40714f0 2023-12-04T05:45:58.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5ca9b16640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ca4070380 0x7f5ca41f11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:58.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5ca9b16640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ca4070380 0x7f5ca41f11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41372/0 (socket says 172.21.15.107:41372) 2023-12-04T05:45:58.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5ca9b16640 1 -- 172.21.15.107:0/3085727353 learned_addr learned my addr 172.21.15.107:0/3085727353 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:45:58.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5caa317640 1 --2- 172.21.15.107:0/3085727353 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ca4151180 0x7f5ca40c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:58.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5ca9315640 1 --2- 172.21.15.107:0/3085727353 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5ca40714f0 0x7f5ca41f1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:58.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5ca9b16640 1 -- 172.21.15.107:0/3085727353 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5ca40714f0 msgr2=0x7f5ca41f1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:58.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5ca9b16640 1 --2- 172.21.15.107:0/3085727353 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5ca40714f0 0x7f5ca41f1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:58.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5ca9b16640 1 -- 172.21.15.107:0/3085727353 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ca4151180 msgr2=0x7f5ca40c4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:58.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5ca9b16640 1 --2- 172.21.15.107:0/3085727353 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ca4151180 0x7f5ca40c4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:58.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5ca9b16640 1 -- 172.21.15.107:0/3085727353 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c940094d0 con 0x7f5ca4070380 2023-12-04T05:45:58.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5caa317640 1 --2- 172.21.15.107:0/3085727353 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ca4151180 0x7f5ca40c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:45:58.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5ca9315640 1 --2- 172.21.15.107:0/3085727353 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5ca40714f0 0x7f5ca41f1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:45:58.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5ca9b16640 1 --2- 172.21.15.107:0/3085727353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ca4070380 0x7f5ca41f11c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5ca000c7f0 tx=0x7f5ca000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:58.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5c9a7fc640 1 -- 172.21.15.107:0/3085727353 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca0007820 con 0x7f5ca4070380 2023-12-04T05:45:58.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5c9a7fc640 1 -- 172.21.15.107:0/3085727353 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca0007980 con 0x7f5ca4070380 2023-12-04T05:45:58.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5cabd7a640 1 -- 172.21.15.107:0/3085727353 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca40c3f70 con 0x7f5ca4070380 2023-12-04T05:45:58.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5c9a7fc640 1 -- 172.21.15.107:0/3085727353 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca0007d30 con 0x7f5ca4070380 2023-12-04T05:45:58.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.290+0000 7f5cabd7a640 1 -- 172.21.15.107:0/3085727353 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca414c5d0 con 0x7f5ca4070380 2023-12-04T05:45:58.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.294+0000 7f5cabd7a640 1 -- 172.21.15.107:0/3085727353 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca4067870 con 0x7f5ca4070380 2023-12-04T05:45:58.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.294+0000 7f5c9a7fc640 1 -- 172.21.15.107:0/3085727353 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f5ca00113f0 con 0x7f5ca4070380 2023-12-04T05:45:58.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.294+0000 7f5c9a7fc640 1 --2- 172.21.15.107:0/3085727353 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5c880668d0 0x7f5c88068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:45:58.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.294+0000 7f5c9a7fc640 1 -- 172.21.15.107:0/3085727353 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f5ca0084910 con 0x7f5ca4070380 2023-12-04T05:45:58.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.294+0000 7f5ca9315640 1 --2- 172.21.15.107:0/3085727353 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5c880668d0 0x7f5c88068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:45:58.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.294+0000 7f5ca9315640 1 --2- 172.21.15.107:0/3085727353 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5c880668d0 0x7f5c88068d90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5c940b2000 tx=0x7f5c940aa070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:45:58.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.298+0000 7f5c9a7fc640 1 -- 172.21.15.107:0/3085727353 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5ca0051970 con 0x7f5ca4070380 2023-12-04T05:45:58.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.478+0000 7f5cabd7a640 1 -- 172.21.15.107:0/3085727353 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ca41509b0 con 0x7f5ca4070380 2023-12-04T05:45:58.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.478+0000 7f5c9a7fc640 1 -- 172.21.15.107:0/3085727353 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4166 (secure 0 0 0) 0x7f5ca001f020 con 0x7f5ca4070380 2023-12-04T05:45:58.481 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:45:58.482 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:54.798865+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4288,"mds_1":5190,"mds_2":4263},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:stopping","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:45:58.482 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 45 2023-12-04T05:45:58.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.482+0000 7f5cabd7a640 1 -- 172.21.15.107:0/3085727353 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5c880668d0 msgr2=0x7f5c88068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:58.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.482+0000 7f5cabd7a640 1 --2- 172.21.15.107:0/3085727353 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5c880668d0 0x7f5c88068d90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5c940b2000 tx=0x7f5c940aa070 comp rx=0 tx=0).stop 2023-12-04T05:45:58.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.482+0000 7f5cabd7a640 1 -- 172.21.15.107:0/3085727353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ca4070380 msgr2=0x7f5ca41f11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:45:58.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.482+0000 7f5cabd7a640 1 --2- 172.21.15.107:0/3085727353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ca4070380 0x7f5ca41f11c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5ca000c7f0 tx=0x7f5ca000ccb0 comp rx=0 tx=0).stop 2023-12-04T05:45:58.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.482+0000 7f5cabd7a640 1 -- 172.21.15.107:0/3085727353 shutdown_connections 2023-12-04T05:45:58.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.482+0000 7f5cabd7a640 1 --2- 172.21.15.107:0/3085727353 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5c880668d0 0x7f5c88068d90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:58.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.482+0000 7f5cabd7a640 1 --2- 172.21.15.107:0/3085727353 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5ca4151180 0x7f5ca40c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:58.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.482+0000 7f5cabd7a640 1 --2- 172.21.15.107:0/3085727353 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5ca40714f0 0x7f5ca41f1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:58.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.482+0000 7f5cabd7a640 1 --2- 172.21.15.107:0/3085727353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ca4070380 0x7f5ca41f11c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:45:58.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.482+0000 7f5cabd7a640 1 -- 172.21.15.107:0/3085727353 >> 172.21.15.107:0/3085727353 conn(0x7f5ca40c7ff0 msgr2=0x7f5ca406d910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:45:58.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.482+0000 7f5cabd7a640 1 -- 172.21.15.107:0/3085727353 shutdown_connections 2023-12-04T05:45:58.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:45:58.482+0000 7f5cabd7a640 1 -- 172.21.15.107:0/3085727353 wait complete. 2023-12-04T05:45:58.499 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:46:00.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:00.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.626+0000 7f397a49a640 1 Processor -- start 2023-12-04T05:46:00.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.630+0000 7f397a49a640 1 -- start start 2023-12-04T05:46:00.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.630+0000 7f397a49a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3974071c00 0x7f39740c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:00.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.630+0000 7f397a49a640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f3974070380 0x7f3974070780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:00.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.630+0000 7f397a49a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39740700c0 con 0x7f3974151180 2023-12-04T05:46:00.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.630+0000 7f397a49a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3974156df0 con 0x7f3974071c00 2023-12-04T05:46:00.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.630+0000 7f397a49a640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39741570d0 con 0x7f3974070380 2023-12-04T05:46:00.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.630+0000 7f39737fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3974071c00 0x7f39740c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:00.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.630+0000 7f3973fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3974151180 0x7f39740714f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58772/0 (socket says 172.21.15.107:58772) 2023-12-04T05:46:00.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.630+0000 7f39737fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3974071c00 0x7f39740c79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41384/0 (socket says 172.21.15.107:41384) 2023-12-04T05:46:00.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.630+0000 7f3973fff640 1 -- 172.21.15.107:0/3808297852 learned_addr learned my addr 172.21.15.107:0/3808297852 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:46:00.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.630+0000 7f39737fe640 1 -- 172.21.15.107:0/3808297852 >> v2:172.21.15.195:3300/0 conn(0x7f3974070380 msgr2=0x7f3974070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:00.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.630+0000 7f39737fe640 1 --2- 172.21.15.107:0/3808297852 >> v2:172.21.15.195:3300/0 conn(0x7f3974070380 0x7f3974070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:00.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.630+0000 7f39737fe640 1 -- 172.21.15.107:0/3808297852 >> v1:172.21.15.107:6789/0 conn(0x7f3974151180 legacy=0x7f39740714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:00.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.630+0000 7f39737fe640 1 -- 172.21.15.107:0/3808297852 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39741573b0 con 0x7f3974071c00 2023-12-04T05:46:00.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.630+0000 7f3972ffd640 1 -- 172.21.15.107:0/3808297852 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563365243 0 0) 0x7f39740700c0 con 0x7f3974151180 2023-12-04T05:46:00.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.630+0000 7f39737fe640 1 --2- 172.21.15.107:0/3808297852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3974071c00 0x7f39740c79e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f397406f680 tx=0x7f395c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc3906ed170a576a server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:00.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.630+0000 7f3972ffd640 1 -- 172.21.15.107:0/3808297852 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395c0a8950 con 0x7f3974071c00 2023-12-04T05:46:00.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.630+0000 7f3972ffd640 1 -- 172.21.15.107:0/3808297852 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395c0a8ab0 con 0x7f3974071c00 2023-12-04T05:46:00.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.630+0000 7f3972ffd640 1 -- 172.21.15.107:0/3808297852 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395c0a8e40 con 0x7f3974071c00 2023-12-04T05:46:00.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f397a49a640 1 -- 172.21.15.107:0/3808297852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3974071c00 msgr2=0x7f39740c79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:00.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f397a49a640 1 --2- 172.21.15.107:0/3808297852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3974071c00 0x7f39740c79e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f397406f680 tx=0x7f395c01ec50 comp rx=0 tx=0).stop 2023-12-04T05:46:00.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f397a49a640 1 -- 172.21.15.107:0/3808297852 shutdown_connections 2023-12-04T05:46:00.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f397a49a640 1 --2- 172.21.15.107:0/3808297852 >> v2:172.21.15.195:3300/0 conn(0x7f3974070380 0x7f3974070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:00.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f397a49a640 1 --2- 172.21.15.107:0/3808297852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3974071c00 0x7f39740c79e0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:00.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f397a49a640 1 -- 172.21.15.107:0/3808297852 >> 172.21.15.107:0/3808297852 conn(0x7f39740c7ff0 msgr2=0x7f39740c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:00.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f397a49a640 1 -- 172.21.15.107:0/3808297852 shutdown_connections 2023-12-04T05:46:00.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f397a49a640 1 -- 172.21.15.107:0/3808297852 wait complete. 2023-12-04T05:46:00.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f397a49a640 1 Processor -- start 2023-12-04T05:46:00.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f397a49a640 1 -- start start 2023-12-04T05:46:00.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f397a49a640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3974070380 0x7f3974140790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:00.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f397a49a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3974071c00 0x7f39741e9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:00.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f397a49a640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3974151180 0x7f3974146e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:00.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f397a49a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39741404d0 con 0x7f3974071c00 2023-12-04T05:46:00.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f397a49a640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397414c040 con 0x7f3974070380 2023-12-04T05:46:00.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f397a49a640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3974140da0 con 0x7f3974151180 2023-12-04T05:46:00.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f39737fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3974071c00 0x7f39741e9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:00.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f39737fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3974071c00 0x7f39741e9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41394/0 (socket says 172.21.15.107:41394) 2023-12-04T05:46:00.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f39737fe640 1 -- 172.21.15.107:0/1603375611 learned_addr learned my addr 172.21.15.107:0/1603375611 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:00.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f39737fe640 1 -- 172.21.15.107:0/1603375611 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3974151180 msgr2=0x7f3974146e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:00.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f39737fe640 1 --2- 172.21.15.107:0/1603375611 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3974151180 0x7f3974146e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:00.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f3973fff640 1 --2- 172.21.15.107:0/1603375611 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3974070380 0x7f3974140790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:00.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f39737fe640 1 -- 172.21.15.107:0/1603375611 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3974070380 msgr2=0x7f3974140790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:00.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f39737fe640 1 --2- 172.21.15.107:0/1603375611 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3974070380 0x7f3974140790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:00.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f39737fe640 1 -- 172.21.15.107:0/1603375611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395c009420 con 0x7f3974071c00 2023-12-04T05:46:00.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f39737fe640 1 --2- 172.21.15.107:0/1603375611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3974071c00 0x7f39741e9f30 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f395c0b3040 tx=0x7f395c01b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:00.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f3970ff9640 1 -- 172.21.15.107:0/1603375611 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395c0b0e80 con 0x7f3974071c00 2023-12-04T05:46:00.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f3970ff9640 1 -- 172.21.15.107:0/1603375611 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395c0b9410 con 0x7f3974071c00 2023-12-04T05:46:00.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f3970ff9640 1 -- 172.21.15.107:0/1603375611 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395c0b97c0 con 0x7f3974071c00 2023-12-04T05:46:00.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f397a49a640 1 -- 172.21.15.107:0/1603375611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3974141060 con 0x7f3974071c00 2023-12-04T05:46:00.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.634+0000 7f397a49a640 1 -- 172.21.15.107:0/1603375611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f397414b710 con 0x7f3974071c00 2023-12-04T05:46:00.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.638+0000 7f397a49a640 1 -- 172.21.15.107:0/1603375611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3974067870 con 0x7f3974071c00 2023-12-04T05:46:00.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.638+0000 7f3970ff9640 1 -- 172.21.15.107:0/1603375611 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f395c0b7040 con 0x7f3974071c00 2023-12-04T05:46:00.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.638+0000 7f3970ff9640 1 --2- 172.21.15.107:0/1603375611 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f39540668d0 0x7f3954068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:00.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.638+0000 7f3970ff9640 1 -- 172.21.15.107:0/1603375611 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f395c1223c0 con 0x7f3974071c00 2023-12-04T05:46:00.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.638+0000 7f3973fff640 1 --2- 172.21.15.107:0/1603375611 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f39540668d0 0x7f3954068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:00.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.638+0000 7f3973fff640 1 --2- 172.21.15.107:0/1603375611 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f39540668d0 0x7f3954068d90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3964000f70 tx=0x7f3964009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:00.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.642+0000 7f3970ff9640 1 -- 172.21.15.107:0/1603375611 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f395c0ef420 con 0x7f3974071c00 2023-12-04T05:46:00.785 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:46:00.789 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:46:00.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.826+0000 7f397a49a640 1 -- 172.21.15.107:0/1603375611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3974073100 con 0x7f3974071c00 2023-12-04T05:46:00.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.826+0000 7f3970ff9640 1 -- 172.21.15.107:0/1603375611 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4167 (secure 0 0 0) 0x7f395c0f2a40 con 0x7f3974071c00 2023-12-04T05:46:00.828 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:00.828 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:59.827803+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4288,"mds_1":5190,"mds_2":4263},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:stopping","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:resolve","state_seq":7,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:00.828 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 46 2023-12-04T05:46:00.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.826+0000 7f397a49a640 1 -- 172.21.15.107:0/1603375611 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f39540668d0 msgr2=0x7f3954068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:00.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.826+0000 7f397a49a640 1 --2- 172.21.15.107:0/1603375611 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f39540668d0 0x7f3954068d90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3964000f70 tx=0x7f3964009040 comp rx=0 tx=0).stop 2023-12-04T05:46:00.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.826+0000 7f397a49a640 1 -- 172.21.15.107:0/1603375611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3974071c00 msgr2=0x7f39741e9f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:00.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.826+0000 7f397a49a640 1 --2- 172.21.15.107:0/1603375611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3974071c00 0x7f39741e9f30 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f395c0b3040 tx=0x7f395c01b9f0 comp rx=0 tx=0).stop 2023-12-04T05:46:00.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.826+0000 7f397a49a640 1 -- 172.21.15.107:0/1603375611 shutdown_connections 2023-12-04T05:46:00.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.826+0000 7f397a49a640 1 --2- 172.21.15.107:0/1603375611 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f39540668d0 0x7f3954068d90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:00.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.826+0000 7f397a49a640 1 --2- 172.21.15.107:0/1603375611 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3974151180 0x7f3974146e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:00.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.826+0000 7f397a49a640 1 --2- 172.21.15.107:0/1603375611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3974071c00 0x7f39741e9f30 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:00.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.826+0000 7f397a49a640 1 --2- 172.21.15.107:0/1603375611 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3974070380 0x7f3974140790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:00.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.826+0000 7f397a49a640 1 -- 172.21.15.107:0/1603375611 >> 172.21.15.107:0/1603375611 conn(0x7f39740c7ff0 msgr2=0x7f39740714f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:00.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.826+0000 7f397a49a640 1 -- 172.21.15.107:0/1603375611 shutdown_connections 2023-12-04T05:46:00.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:00.826+0000 7f397a49a640 1 -- 172.21.15.107:0/1603375611 wait complete. 2023-12-04T05:46:00.845 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:46:01.579 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~20s 2023-12-04T05:46:01.579 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~134s 2023-12-04T05:46:01.579 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~175s 2023-12-04T05:46:02.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:02.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.974+0000 7fc71064e640 1 Processor -- start 2023-12-04T05:46:02.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.974+0000 7fc71064e640 1 -- start start 2023-12-04T05:46:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc71064e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc7080724f0 0x7fc708158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc71064e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc7080c56f0 0x7fc708140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc71064e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc708158da0 con 0x7fc708072110 2023-12-04T05:46:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc71064e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7081583d0 con 0x7fc7080724f0 2023-12-04T05:46:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc71064e640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7080c6170 con 0x7fc7080c56f0 2023-12-04T05:46:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc70dbe9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc7080724f0 0x7fc708158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc70dbe9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc7080724f0 0x7fc708158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41408/0 (socket says 172.21.15.107:41408) 2023-12-04T05:46:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc70dbe9640 1 -- 172.21.15.107:0/486967846 learned_addr learned my addr 172.21.15.107:0/486967846 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc70dbe9640 1 -- 172.21.15.107:0/486967846 >> v2:172.21.15.195:3300/0 conn(0x7fc7080c56f0 msgr2=0x7fc708140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc70dbe9640 1 --2- 172.21.15.107:0/486967846 >> v2:172.21.15.195:3300/0 conn(0x7fc7080c56f0 0x7fc708140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc70dbe9640 1 -- 172.21.15.107:0/486967846 >> v1:172.21.15.107:6789/0 conn(0x7fc708072110 legacy=0x7fc7080c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc70dbe9640 1 -- 172.21.15.107:0/486967846 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc708157da0 con 0x7fc7080724f0 2023-12-04T05:46:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc70d3e8640 1 -- 172.21.15.107:0/486967846 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094320203 0 0) 0x7fc704001120 con 0x7fc708072110 2023-12-04T05:46:02.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc70dbe9640 1 --2- 172.21.15.107:0/486967846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7080724f0 0x7fc708158860 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc7080c9080 tx=0x7fc6f801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25726f70621b3015 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:02.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc70d3e8640 1 -- 172.21.15.107:0/486967846 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f80a89f0 con 0x7fc7080724f0 2023-12-04T05:46:02.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc70d3e8640 1 -- 172.21.15.107:0/486967846 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f80a8b50 con 0x7fc7080724f0 2023-12-04T05:46:02.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc70d3e8640 1 -- 172.21.15.107:0/486967846 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f80b0440 con 0x7fc7080724f0 2023-12-04T05:46:02.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc71064e640 1 -- 172.21.15.107:0/486967846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7080724f0 msgr2=0x7fc708158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:02.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc71064e640 1 --2- 172.21.15.107:0/486967846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7080724f0 0x7fc708158860 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc7080c9080 tx=0x7fc6f801ecf0 comp rx=0 tx=0).stop 2023-12-04T05:46:02.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc71064e640 1 -- 172.21.15.107:0/486967846 shutdown_connections 2023-12-04T05:46:02.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc71064e640 1 --2- 172.21.15.107:0/486967846 >> v2:172.21.15.195:3300/0 conn(0x7fc7080c56f0 0x7fc708140830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:02.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc71064e640 1 --2- 172.21.15.107:0/486967846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7080724f0 0x7fc708158860 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:02.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.978+0000 7fc71064e640 1 -- 172.21.15.107:0/486967846 >> 172.21.15.107:0/486967846 conn(0x7fc7080c7ff0 msgr2=0x7fc7080c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:02.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc71064e640 1 -- 172.21.15.107:0/486967846 shutdown_connections 2023-12-04T05:46:02.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc71064e640 1 -- 172.21.15.107:0/486967846 wait complete. 2023-12-04T05:46:02.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc71064e640 1 Processor -- start 2023-12-04T05:46:02.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc71064e640 1 -- start start 2023-12-04T05:46:02.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc71064e640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc708072110 0x7fc708149970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:02.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc71064e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7080724f0 0x7fc7080c7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:02.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc71064e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc7080c56f0 0x7fc7081f3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:02.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc71064e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc708149eb0 con 0x7fc7080724f0 2023-12-04T05:46:02.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc71064e640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7080711b0 con 0x7fc708072110 2023-12-04T05:46:02.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc71064e640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc70806fdd0 con 0x7fc7080c56f0 2023-12-04T05:46:02.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc70dbe9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7080724f0 0x7fc7080c7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:02.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc70dbe9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7080724f0 0x7fc7080c7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41412/0 (socket says 172.21.15.107:41412) 2023-12-04T05:46:02.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc70dbe9640 1 -- 172.21.15.107:0/3919954889 learned_addr learned my addr 172.21.15.107:0/3919954889 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:02.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc70e3ea640 1 --2- 172.21.15.107:0/3919954889 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc708072110 0x7fc708149970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:02.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc70dbe9640 1 -- 172.21.15.107:0/3919954889 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc7080c56f0 msgr2=0x7fc7081f3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:02.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc70ebeb640 1 --2- 172.21.15.107:0/3919954889 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc7080c56f0 0x7fc7081f3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:02.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc70dbe9640 1 --2- 172.21.15.107:0/3919954889 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc7080c56f0 0x7fc7081f3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:02.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc70dbe9640 1 -- 172.21.15.107:0/3919954889 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc708072110 msgr2=0x7fc708149970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:02.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc70dbe9640 1 --2- 172.21.15.107:0/3919954889 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc708072110 0x7fc708149970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:02.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc70dbe9640 1 -- 172.21.15.107:0/3919954889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f80094d0 con 0x7fc7080724f0 2023-12-04T05:46:02.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc70e3ea640 1 --2- 172.21.15.107:0/3919954889 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc708072110 0x7fc708149970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:46:02.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc70dbe9640 1 --2- 172.21.15.107:0/3919954889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7080724f0 0x7fc7080c7450 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc6f80b3040 tx=0x7fc6f801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:02.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc6feffd640 1 -- 172.21.15.107:0/3919954889 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f80b0e00 con 0x7fc7080724f0 2023-12-04T05:46:02.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc6feffd640 1 -- 172.21.15.107:0/3919954889 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f80bb470 con 0x7fc7080724f0 2023-12-04T05:46:02.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc71064e640 1 -- 172.21.15.107:0/3919954889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7080700b0 con 0x7fc7080724f0 2023-12-04T05:46:02.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc6feffd640 1 -- 172.21.15.107:0/3919954889 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f80bb820 con 0x7fc7080724f0 2023-12-04T05:46:02.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.982+0000 7fc71064e640 1 -- 172.21.15.107:0/3919954889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc708071750 con 0x7fc7080724f0 2023-12-04T05:46:02.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.986+0000 7fc71064e640 1 -- 172.21.15.107:0/3919954889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc708067870 con 0x7fc7080724f0 2023-12-04T05:46:02.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.986+0000 7fc6feffd640 1 -- 172.21.15.107:0/3919954889 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fc6f80b7070 con 0x7fc7080724f0 2023-12-04T05:46:02.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.986+0000 7fc6feffd640 1 --2- 172.21.15.107:0/3919954889 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc6ec0668d0 0x7fc6ec068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:02.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.986+0000 7fc6feffd640 1 -- 172.21.15.107:0/3919954889 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fc6f8122a60 con 0x7fc7080724f0 2023-12-04T05:46:02.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.986+0000 7fc70e3ea640 1 --2- 172.21.15.107:0/3919954889 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc6ec0668d0 0x7fc6ec068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:02.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.986+0000 7fc70e3ea640 1 --2- 172.21.15.107:0/3919954889 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc6ec0668d0 0x7fc6ec068d90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc704000f70 tx=0x7fc704009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:02.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:02.990+0000 7fc6feffd640 1 -- 172.21.15.107:0/3919954889 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc6f80efad0 con 0x7fc7080724f0 2023-12-04T05:46:03.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:03.178+0000 7fc71064e640 1 -- 172.21.15.107:0/3919954889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7080c6a10 con 0x7fc7080724f0 2023-12-04T05:46:03.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:03.178+0000 7fc6feffd640 1 -- 172.21.15.107:0/3919954889 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4167 (secure 0 0 0) 0x7fc6f80b5020 con 0x7fc7080724f0 2023-12-04T05:46:03.180 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:03.181 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:59.827803+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4288,"mds_1":5190,"mds_2":4263},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:stopping","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:resolve","state_seq":7,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:03.181 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 46 2023-12-04T05:46:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:03.178+0000 7fc71064e640 1 -- 172.21.15.107:0/3919954889 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc6ec0668d0 msgr2=0x7fc6ec068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:03.178+0000 7fc71064e640 1 --2- 172.21.15.107:0/3919954889 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc6ec0668d0 0x7fc6ec068d90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc704000f70 tx=0x7fc704009040 comp rx=0 tx=0).stop 2023-12-04T05:46:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:03.178+0000 7fc71064e640 1 -- 172.21.15.107:0/3919954889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7080724f0 msgr2=0x7fc7080c7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:03.178+0000 7fc71064e640 1 --2- 172.21.15.107:0/3919954889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7080724f0 0x7fc7080c7450 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc6f80b3040 tx=0x7fc6f801ba90 comp rx=0 tx=0).stop 2023-12-04T05:46:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:03.178+0000 7fc71064e640 1 -- 172.21.15.107:0/3919954889 shutdown_connections 2023-12-04T05:46:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:03.178+0000 7fc71064e640 1 --2- 172.21.15.107:0/3919954889 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc6ec0668d0 0x7fc6ec068d90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:03.178+0000 7fc71064e640 1 --2- 172.21.15.107:0/3919954889 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc7080c56f0 0x7fc7081f3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:03.178+0000 7fc71064e640 1 --2- 172.21.15.107:0/3919954889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7080724f0 0x7fc7080c7450 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:03.178+0000 7fc71064e640 1 --2- 172.21.15.107:0/3919954889 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc708072110 0x7fc708149970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:03.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:03.178+0000 7fc71064e640 1 -- 172.21.15.107:0/3919954889 >> 172.21.15.107:0/3919954889 conn(0x7fc7080c7ff0 msgr2=0x7fc708156bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:03.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:03.182+0000 7fc71064e640 1 -- 172.21.15.107:0/3919954889 shutdown_connections 2023-12-04T05:46:03.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:03.182+0000 7fc71064e640 1 -- 172.21.15.107:0/3919954889 wait complete. 2023-12-04T05:46:03.199 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:46:05.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:05.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.330+0000 7f4e67f31640 1 Processor -- start 2023-12-04T05:46:05.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.330+0000 7f4e67f31640 1 -- start start 2023-12-04T05:46:05.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.334+0000 7f4e67f31640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4e600c7540 0x7f4e600725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:05.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.334+0000 7f4e67f31640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e60158b80 con 0x7f4e600ca2e0 2023-12-04T05:46:05.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.334+0000 7f4e67f31640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e60157f70 con 0x7f4e600ca6c0 2023-12-04T05:46:05.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.334+0000 7f4e67f31640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e60158250 con 0x7f4e600c7540 2023-12-04T05:46:05.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.334+0000 7f4e65ccd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4e600ca2e0 0x7f4e600c6e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58794/0 (socket says 172.21.15.107:58794) 2023-12-04T05:46:05.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.334+0000 7f4e664ce640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4e600c7540 0x7f4e600725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:05.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.334+0000 7f4e65ccd640 1 -- 172.21.15.107:0/4213531376 learned_addr learned my addr 172.21.15.107:0/4213531376 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:46:05.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.334+0000 7f4e664ce640 1 -- 172.21.15.107:0/4213531376 >> v1:172.21.15.195:6789/0 conn(0x7f4e600ca6c0 legacy=0x7f4e60151990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:05.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.334+0000 7f4e664ce640 1 -- 172.21.15.107:0/4213531376 >> v1:172.21.15.107:6789/0 conn(0x7f4e600ca2e0 legacy=0x7f4e600c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:05.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.334+0000 7f4e664ce640 1 -- 172.21.15.107:0/4213531376 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e60158530 con 0x7f4e600c7540 2023-12-04T05:46:05.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.334+0000 7f4e664ce640 1 --2- 172.21.15.107:0/4213531376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e600c7540 0x7f4e600725c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4e58009d90 tx=0x7f4e5801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec4a08ec3036dcd server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:05.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.334+0000 7f4e64ccb640 1 -- 172.21.15.107:0/4213531376 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e580a59c0 con 0x7f4e600c7540 2023-12-04T05:46:05.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.334+0000 7f4e64ccb640 1 -- 172.21.15.107:0/4213531376 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e580a5b20 con 0x7f4e600c7540 2023-12-04T05:46:05.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.334+0000 7f4e64ccb640 1 -- 172.21.15.107:0/4213531376 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e580ad440 con 0x7f4e600c7540 2023-12-04T05:46:05.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.334+0000 7f4e67f31640 1 -- 172.21.15.107:0/4213531376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e600c7540 msgr2=0x7f4e600725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:05.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.334+0000 7f4e67f31640 1 --2- 172.21.15.107:0/4213531376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e600c7540 0x7f4e600725c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4e58009d90 tx=0x7f4e5801c8e0 comp rx=0 tx=0).stop 2023-12-04T05:46:05.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.334+0000 7f4e67f31640 1 -- 172.21.15.107:0/4213531376 shutdown_connections 2023-12-04T05:46:05.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.334+0000 7f4e67f31640 1 --2- 172.21.15.107:0/4213531376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e600c7540 0x7f4e600725c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:05.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.334+0000 7f4e67f31640 1 -- 172.21.15.107:0/4213531376 >> 172.21.15.107:0/4213531376 conn(0x7f4e600663a0 msgr2=0x7f4e600667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:05.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.334+0000 7f4e67f31640 1 -- 172.21.15.107:0/4213531376 shutdown_connections 2023-12-04T05:46:05.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.334+0000 7f4e67f31640 1 -- 172.21.15.107:0/4213531376 wait complete. 2023-12-04T05:46:05.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e67f31640 1 Processor -- start 2023-12-04T05:46:05.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e67f31640 1 -- start start 2023-12-04T05:46:05.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e67f31640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4e600c7540 0x7f4e60154840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:05.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e67f31640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4e600ca2e0 0x7f4e600cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:05.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e67f31640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e600ca6c0 0x7f4e601539b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:05.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e67f31640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e60152670 con 0x7f4e600ca6c0 2023-12-04T05:46:05.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e67f31640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e600c4560 con 0x7f4e600c7540 2023-12-04T05:46:05.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e67f31640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e60152910 con 0x7f4e600ca2e0 2023-12-04T05:46:05.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e664ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e600ca6c0 0x7f4e601539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:05.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e664ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e600ca6c0 0x7f4e601539b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41436/0 (socket says 172.21.15.107:41436) 2023-12-04T05:46:05.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e664ce640 1 -- 172.21.15.107:0/3734263143 learned_addr learned my addr 172.21.15.107:0/3734263143 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:05.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e65ccd640 1 --2- 172.21.15.107:0/3734263143 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4e600c7540 0x7f4e60154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:05.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e664ce640 1 -- 172.21.15.107:0/3734263143 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4e600ca2e0 msgr2=0x7f4e600cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:05.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e664ce640 1 --2- 172.21.15.107:0/3734263143 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4e600ca2e0 0x7f4e600cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:05.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e664ce640 1 -- 172.21.15.107:0/3734263143 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4e600c7540 msgr2=0x7f4e60154840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:05.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e664ce640 1 --2- 172.21.15.107:0/3734263143 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4e600c7540 0x7f4e60154840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:05.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e664ce640 1 -- 172.21.15.107:0/3734263143 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e58009a70 con 0x7f4e600ca6c0 2023-12-04T05:46:05.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e664ce640 1 --2- 172.21.15.107:0/3734263143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e600ca6c0 0x7f4e601539b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4e580b0040 tx=0x7f4e58019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:05.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e567fc640 1 -- 172.21.15.107:0/3734263143 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e580ade40 con 0x7f4e600ca6c0 2023-12-04T05:46:05.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e567fc640 1 -- 172.21.15.107:0/3734263143 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e580b9450 con 0x7f4e600ca6c0 2023-12-04T05:46:05.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e67f31640 1 -- 172.21.15.107:0/3734263143 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e60152bf0 con 0x7f4e600ca6c0 2023-12-04T05:46:05.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e567fc640 1 -- 172.21.15.107:0/3734263143 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e580b9800 con 0x7f4e600ca6c0 2023-12-04T05:46:05.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e67f31640 1 -- 172.21.15.107:0/3734263143 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e601531b0 con 0x7f4e600ca6c0 2023-12-04T05:46:05.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.338+0000 7f4e67f31640 1 -- 172.21.15.107:0/3734263143 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e60067ec0 con 0x7f4e600ca6c0 2023-12-04T05:46:05.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.342+0000 7f4e567fc640 1 -- 172.21.15.107:0/3734263143 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4e580b9960 con 0x7f4e600ca6c0 2023-12-04T05:46:05.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.342+0000 7f4e567fc640 1 --2- 172.21.15.107:0/3734263143 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4e440668d0 0x7f4e44068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:05.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.342+0000 7f4e567fc640 1 -- 172.21.15.107:0/3734263143 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f4e5811fdb0 con 0x7f4e600ca6c0 2023-12-04T05:46:05.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.342+0000 7f4e65ccd640 1 --2- 172.21.15.107:0/3734263143 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4e440668d0 0x7f4e44068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:05.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.342+0000 7f4e65ccd640 1 --2- 172.21.15.107:0/3734263143 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4e440668d0 0x7f4e44068d90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c005890 tx=0x7f4e5c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:05.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.346+0000 7f4e567fc640 1 -- 172.21.15.107:0/3734263143 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4e580ecf70 con 0x7f4e600ca6c0 2023-12-04T05:46:05.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.530+0000 7f4e67f31640 1 -- 172.21.15.107:0/3734263143 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e60072510 con 0x7f4e600ca6c0 2023-12-04T05:46:05.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.534+0000 7f4e567fc640 1 -- 172.21.15.107:0/3734263143 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4167 (secure 0 0 0) 0x7f4e580f0590 con 0x7f4e600ca6c0 2023-12-04T05:46:05.535 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:05.535 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:45:59.827803+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4288,"mds_1":5190,"mds_2":4263},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:stopping","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:resolve","state_seq":7,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:05.535 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 46 2023-12-04T05:46:05.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.534+0000 7f4e67f31640 1 -- 172.21.15.107:0/3734263143 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4e440668d0 msgr2=0x7f4e44068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:05.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.534+0000 7f4e67f31640 1 --2- 172.21.15.107:0/3734263143 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4e440668d0 0x7f4e44068d90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c005890 tx=0x7f4e5c009040 comp rx=0 tx=0).stop 2023-12-04T05:46:05.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.534+0000 7f4e67f31640 1 -- 172.21.15.107:0/3734263143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e600ca6c0 msgr2=0x7f4e601539b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:05.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.534+0000 7f4e67f31640 1 --2- 172.21.15.107:0/3734263143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e600ca6c0 0x7f4e601539b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4e580b0040 tx=0x7f4e58019680 comp rx=0 tx=0).stop 2023-12-04T05:46:05.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.534+0000 7f4e67f31640 1 -- 172.21.15.107:0/3734263143 shutdown_connections 2023-12-04T05:46:05.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.534+0000 7f4e67f31640 1 --2- 172.21.15.107:0/3734263143 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4e440668d0 0x7f4e44068d90 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:05.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.534+0000 7f4e67f31640 1 --2- 172.21.15.107:0/3734263143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e600ca6c0 0x7f4e601539b0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:05.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.534+0000 7f4e67f31640 1 --2- 172.21.15.107:0/3734263143 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4e600ca2e0 0x7f4e600cc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:05.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.534+0000 7f4e67f31640 1 --2- 172.21.15.107:0/3734263143 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4e600c7540 0x7f4e60154840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:05.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.534+0000 7f4e67f31640 1 -- 172.21.15.107:0/3734263143 >> 172.21.15.107:0/3734263143 conn(0x7f4e600663a0 msgr2=0x7f4e600c6e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:05.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.534+0000 7f4e67f31640 1 -- 172.21.15.107:0/3734263143 shutdown_connections 2023-12-04T05:46:05.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:05.534+0000 7f4e67f31640 1 -- 172.21.15.107:0/3734263143 wait complete. 2023-12-04T05:46:05.553 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-12-04T05:46:05.553 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 46, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-12-04T05:39:59.032019+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 46, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4263": { "addr": "172.21.15.107:6837/2210527429", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6835", "nonce": 2210527429, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 2210527429, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4263, "incarnation": 12, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:stopping", "state_seq": 7 }, "gid_4288": { "addr": "172.21.15.195:6837/3412865477", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6835", "nonce": 3412865477, "type": "v2" }, { "addr": "172.21.15.195:6837", "nonce": 3412865477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_5190": { "addr": "172.21.15.107:6836/2928623624", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 2928623624, "type": "v2" }, { "addr": "172.21.15.107:6836", "nonce": 2928623624, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5190, "incarnation": 45, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:resolve", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-12-04T05:45:59.827803+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 3, 4 ], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 5190, "mds_2": 4263 } } } ], "standbys": [] } 2023-12-04T05:46:07.554 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:07.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a9c17640 1 Processor -- start 2023-12-04T05:46:07.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a9c17640 1 -- start start 2023-12-04T05:46:07.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a9c17640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f53a40724f0 0x7f53a4158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:07.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a9c17640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a4158da0 con 0x7f53a4072110 2023-12-04T05:46:07.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a9c17640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a41583d0 con 0x7f53a40c56f0 2023-12-04T05:46:07.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a9c17640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53a40c6170 con 0x7f53a40724f0 2023-12-04T05:46:07.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a37fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f53a4072110 0x7f53a40c7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55178/0 (socket says 172.21.15.107:55178) 2023-12-04T05:46:07.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a37fe640 1 -- 172.21.15.107:0/1904916754 learned_addr learned my addr 172.21.15.107:0/1904916754 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:46:07.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f539affd640 1 --2- 172.21.15.107:0/1904916754 >> v2:172.21.15.195:3300/0 conn(0x7f53a40724f0 0x7f53a4158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:07.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a2ffd640 1 -- 172.21.15.107:0/1904916754 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734690070 0 0) 0x7f5394001120 con 0x7f53a4072110 2023-12-04T05:46:07.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a2ffd640 1 -- 172.21.15.107:0/1904916754 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5390003530 con 0x7f53a4072110 2023-12-04T05:46:07.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a2ffd640 1 -- 172.21.15.107:0/1904916754 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1082818644 0 0) 0x7f5390003530 con 0x7f53a4072110 2023-12-04T05:46:07.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a2ffd640 1 -- 172.21.15.107:0/1904916754 >> v2:172.21.15.195:3300/0 conn(0x7f53a40724f0 msgr2=0x7f53a4158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:07.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a2ffd640 1 --2- 172.21.15.107:0/1904916754 >> v2:172.21.15.195:3300/0 conn(0x7f53a40724f0 0x7f53a4158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:07.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a2ffd640 1 -- 172.21.15.107:0/1904916754 >> v1:172.21.15.195:6789/0 conn(0x7f53a40c56f0 legacy=0x7f53a4140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:07.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a2ffd640 1 -- 172.21.15.107:0/1904916754 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a4157da0 con 0x7f53a4072110 2023-12-04T05:46:07.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a2ffd640 1 -- 172.21.15.107:0/1904916754 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f5394001810 con 0x7f53a4072110 2023-12-04T05:46:07.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a2ffd640 1 -- 172.21.15.107:0/1904916754 >> v1:172.21.15.107:6789/0 conn(0x7f53a4072110 legacy=0x7f53a40c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:07.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a2ffd640 1 --2- 172.21.15.107:0/1904916754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53900039d0 0x7f5390003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:07.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a2ffd640 1 --2- 172.21.15.107:0/1904916754 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f53900047d0 0x7f5390004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:07.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a2ffd640 1 --2- 172.21.15.107:0/1904916754 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5390005190 0x7f539000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:07.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a2ffd640 1 -- 172.21.15.107:0/1904916754 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a40c6170 con 0x7f53900039d0 2023-12-04T05:46:07.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a2ffd640 1 -- 172.21.15.107:0/1904916754 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539000c070 con 0x7f53900047d0 2023-12-04T05:46:07.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a2ffd640 1 -- 172.21.15.107:0/1904916754 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539000c230 con 0x7f5390005190 2023-12-04T05:46:07.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a37fe640 1 --2- 172.21.15.107:0/1904916754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53900039d0 0x7f5390003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:07.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a37fe640 1 -- 172.21.15.107:0/1904916754 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5390005190 msgr2=0x7f539000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:07.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a3fff640 1 --2- 172.21.15.107:0/1904916754 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5390005190 0x7f539000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:07.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f539affd640 1 --2- 172.21.15.107:0/1904916754 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f53900047d0 0x7f5390004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:07.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a37fe640 1 --2- 172.21.15.107:0/1904916754 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f5390005190 0x7f539000b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:07.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a37fe640 1 -- 172.21.15.107:0/1904916754 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f53900047d0 msgr2=0x7f5390004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:07.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a37fe640 1 --2- 172.21.15.107:0/1904916754 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f53900047d0 0x7f5390004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:07.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a37fe640 1 -- 172.21.15.107:0/1904916754 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5390003530 con 0x7f53900039d0 2023-12-04T05:46:07.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.682+0000 7f53a3fff640 1 -- 172.21.15.107:0/1904916754 reap_dead start 2023-12-04T05:46:07.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f53a37fe640 1 --2- 172.21.15.107:0/1904916754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53900039d0 0x7f5390003df0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f539400c0e0 tx=0x7f539401f2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efa22ab740bfc5d6 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:07.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f53a2ffd640 1 -- 172.21.15.107:0/1904916754 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53940058e0 con 0x7f53900039d0 2023-12-04T05:46:07.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f53a2ffd640 1 -- 172.21.15.107:0/1904916754 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5394005a40 con 0x7f53900039d0 2023-12-04T05:46:07.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f53a2ffd640 1 -- 172.21.15.107:0/1904916754 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5394005df0 con 0x7f53900039d0 2023-12-04T05:46:07.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f53a9c17640 1 -- 172.21.15.107:0/1904916754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53900039d0 msgr2=0x7f5390003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:07.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f53a9c17640 1 --2- 172.21.15.107:0/1904916754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53900039d0 0x7f5390003df0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f539400c0e0 tx=0x7f539401f2b0 comp rx=0 tx=0).stop 2023-12-04T05:46:07.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f53a9c17640 1 -- 172.21.15.107:0/1904916754 shutdown_connections 2023-12-04T05:46:07.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f53a9c17640 1 --2- 172.21.15.107:0/1904916754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53900039d0 0x7f5390003df0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:07.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f53a9c17640 1 -- 172.21.15.107:0/1904916754 >> 172.21.15.107:0/1904916754 conn(0x7f53a40c7ff0 msgr2=0x7f53a40c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:07.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f53a9c17640 1 -- 172.21.15.107:0/1904916754 shutdown_connections 2023-12-04T05:46:07.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f53a9c17640 1 -- 172.21.15.107:0/1904916754 wait complete. 2023-12-04T05:46:07.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f53a9c17640 1 Processor -- start 2023-12-04T05:46:07.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f53a9c17640 1 -- start start 2023-12-04T05:46:07.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f53a9c17640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f53900039d0 0x7f53a41ee1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:07.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f53a9c17640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53a414a8b0 0x7f53a4156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:07.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f53a9c17640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f53a415ae70 0x7f53a4149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:07.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f53a9c17640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a41ee6e0 con 0x7f53a414a8b0 2023-12-04T05:46:07.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f53a9c17640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a415a9e0 con 0x7f53a415ae70 2023-12-04T05:46:07.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f53a9c17640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a415c070 con 0x7f53900039d0 2023-12-04T05:46:07.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f539affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53a414a8b0 0x7f53a4156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:07.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f539affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53a414a8b0 0x7f53a4156d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60852/0 (socket says 172.21.15.107:60852) 2023-12-04T05:46:07.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f539affd640 1 -- 172.21.15.107:0/431782472 learned_addr learned my addr 172.21.15.107:0/431782472 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:07.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f53a3fff640 1 --2- 172.21.15.107:0/431782472 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f53a415ae70 0x7f53a4149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:07.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f53a37fe640 1 --2- 172.21.15.107:0/431782472 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f53900039d0 0x7f53a41ee1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:07.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f539affd640 1 -- 172.21.15.107:0/431782472 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f53900039d0 msgr2=0x7f53a41ee1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:07.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f539affd640 1 --2- 172.21.15.107:0/431782472 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f53900039d0 0x7f53a41ee1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:07.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f539affd640 1 -- 172.21.15.107:0/431782472 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f53a415ae70 msgr2=0x7f53a4149580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:07.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f539affd640 1 --2- 172.21.15.107:0/431782472 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f53a415ae70 0x7f53a4149580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:07.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f539affd640 1 -- 172.21.15.107:0/431782472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a415bab0 con 0x7f53a414a8b0 2023-12-04T05:46:07.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.686+0000 7f53a3fff640 1 --2- 172.21.15.107:0/431782472 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f53a415ae70 0x7f53a4149580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:46:07.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.690+0000 7f539affd640 1 --2- 172.21.15.107:0/431782472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53a414a8b0 0x7f53a4156d80 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f538c010790 tx=0x7f538c010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:07.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.690+0000 7f53a0ff9640 1 -- 172.21.15.107:0/431782472 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c00eb00 con 0x7f53a414a8b0 2023-12-04T05:46:07.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.690+0000 7f53a0ff9640 1 -- 172.21.15.107:0/431782472 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538c00ec60 con 0x7f53a414a8b0 2023-12-04T05:46:07.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.690+0000 7f53a9c17640 1 -- 172.21.15.107:0/431782472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a415bd90 con 0x7f53a414a8b0 2023-12-04T05:46:07.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.690+0000 7f53a0ff9640 1 -- 172.21.15.107:0/431782472 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c010ea0 con 0x7f53a414a8b0 2023-12-04T05:46:07.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.690+0000 7f53a9c17640 1 -- 172.21.15.107:0/431782472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a415a670 con 0x7f53a414a8b0 2023-12-04T05:46:07.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.690+0000 7f53a9c17640 1 -- 172.21.15.107:0/431782472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a415a800 con 0x7f53a414a8b0 2023-12-04T05:46:07.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.690+0000 7f53a0ff9640 1 -- 172.21.15.107:0/431782472 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f538c01c400 con 0x7f53a414a8b0 2023-12-04T05:46:07.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.690+0000 7f53a0ff9640 1 --2- 172.21.15.107:0/431782472 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f53840668d0 0x7f5384068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:07.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.690+0000 7f53a37fe640 1 --2- 172.21.15.107:0/431782472 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f53840668d0 0x7f5384068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:07.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.690+0000 7f53a0ff9640 1 -- 172.21.15.107:0/431782472 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f538c0849e0 con 0x7f53a414a8b0 2023-12-04T05:46:07.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.690+0000 7f53a37fe640 1 --2- 172.21.15.107:0/431782472 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f53840668d0 0x7f5384068d90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f53940ae000 tx=0x7f53940aa070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:07.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.694+0000 7f53a0ff9640 1 -- 172.21.15.107:0/431782472 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f538c051a70 con 0x7f53a414a8b0 2023-12-04T05:46:07.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.870+0000 7f53a9c17640 1 -- 172.21.15.107:0/431782472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53a4067870 con 0x7f53a414a8b0 2023-12-04T05:46:07.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.870+0000 7f53a0ff9640 1 -- 172.21.15.107:0/431782472 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4169 (secure 0 0 0) 0x7f538c055090 con 0x7f53a414a8b0 2023-12-04T05:46:07.874 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:07.874 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:05.813739+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4288,"mds_1":5190,"mds_2":4263},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4263":{"gid":4263,"name":"c","rank":2,"incarnation":12,"state":"up:stopping","state_seq":7,"addr":"172.21.15.107:6837/2210527429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2210527429},{"type":"v1","addr":"172.21.15.107:6837","nonce":2210527429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:07.874 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 47 2023-12-04T05:46:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.874+0000 7f53a9c17640 1 -- 172.21.15.107:0/431782472 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f53840668d0 msgr2=0x7f5384068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.874+0000 7f53a9c17640 1 --2- 172.21.15.107:0/431782472 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f53840668d0 0x7f5384068d90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f53940ae000 tx=0x7f53940aa070 comp rx=0 tx=0).stop 2023-12-04T05:46:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.874+0000 7f53a9c17640 1 -- 172.21.15.107:0/431782472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53a414a8b0 msgr2=0x7f53a4156d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.874+0000 7f53a9c17640 1 --2- 172.21.15.107:0/431782472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53a414a8b0 0x7f53a4156d80 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f538c010790 tx=0x7f538c010c50 comp rx=0 tx=0).stop 2023-12-04T05:46:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.874+0000 7f53a9c17640 1 -- 172.21.15.107:0/431782472 shutdown_connections 2023-12-04T05:46:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.874+0000 7f53a9c17640 1 --2- 172.21.15.107:0/431782472 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f53840668d0 0x7f5384068d90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.874+0000 7f53a9c17640 1 --2- 172.21.15.107:0/431782472 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f53a415ae70 0x7f53a4149580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.874+0000 7f53a9c17640 1 --2- 172.21.15.107:0/431782472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53a414a8b0 0x7f53a4156d80 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.874+0000 7f53a9c17640 1 --2- 172.21.15.107:0/431782472 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f53900039d0 0x7f53a41ee1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.874+0000 7f53a9c17640 1 -- 172.21.15.107:0/431782472 >> 172.21.15.107:0/431782472 conn(0x7f53a40c7ff0 msgr2=0x7f53a40c6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.874+0000 7f53a9c17640 1 -- 172.21.15.107:0/431782472 shutdown_connections 2023-12-04T05:46:07.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:07.874+0000 7f53a9c17640 1 -- 172.21.15.107:0/431782472 wait complete. 2023-12-04T05:46:07.891 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:46:08.193 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~26s 2023-12-04T05:46:08.193 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~141s 2023-12-04T05:46:08.193 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~181s 2023-12-04T05:46:09.150 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-12-04T05:46:09.892 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:10.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.022+0000 7f1798dac640 1 Processor -- start 2023-12-04T05:46:10.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.022+0000 7f1798dac640 1 -- start start 2023-12-04T05:46:10.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f1798dac640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f179406e1f0 0x7f179406e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:10.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f1798dac640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1794152360 con 0x7f1794071030 2023-12-04T05:46:10.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f1798dac640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1794071460 con 0x7f1794151f80 2023-12-04T05:46:10.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f1798dac640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f179406acf0 con 0x7f179406e1f0 2023-12-04T05:46:10.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f1791d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1794071030 0x7f17940c5ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55188/0 (socket says 172.21.15.107:55188) 2023-12-04T05:46:10.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f1791d9b640 1 -- 172.21.15.107:0/3627893411 learned_addr learned my addr 172.21.15.107:0/3627893411 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:46:10.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f1792d9d640 1 --2- 172.21.15.107:0/3627893411 >> v2:172.21.15.195:3300/0 conn(0x7f179406e1f0 0x7f179406e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:10.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f179159a640 1 -- 172.21.15.107:0/3627893411 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965735808 0 0) 0x7f1794152360 con 0x7f1794071030 2023-12-04T05:46:10.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f179159a640 1 -- 172.21.15.107:0/3627893411 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1780003530 con 0x7f1794071030 2023-12-04T05:46:10.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f179159a640 1 -- 172.21.15.107:0/3627893411 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3168364823 0 0) 0x7f1780003530 con 0x7f1794071030 2023-12-04T05:46:10.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f179159a640 1 -- 172.21.15.107:0/3627893411 >> v2:172.21.15.195:3300/0 conn(0x7f179406e1f0 msgr2=0x7f179406e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:10.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f179159a640 1 --2- 172.21.15.107:0/3627893411 >> v2:172.21.15.195:3300/0 conn(0x7f179406e1f0 0x7f179406e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:10.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f179159a640 1 -- 172.21.15.107:0/3627893411 >> v1:172.21.15.195:6789/0 conn(0x7f1794151f80 legacy=0x7f179406c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f179159a640 1 -- 172.21.15.107:0/3627893411 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1794157c90 con 0x7f1794071030 2023-12-04T05:46:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f179159a640 1 -- 172.21.15.107:0/3627893411 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f1784001a00 con 0x7f1794071030 2023-12-04T05:46:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f179159a640 1 -- 172.21.15.107:0/3627893411 >> v1:172.21.15.107:6789/0 conn(0x7f1794071030 legacy=0x7f17940c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f179159a640 1 --2- 172.21.15.107:0/3627893411 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1780003980 0x7f1780003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f179159a640 1 --2- 172.21.15.107:0/3627893411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17800047d0 0x7f1780004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f179159a640 1 --2- 172.21.15.107:0/3627893411 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1780005190 0x7f178000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f179159a640 1 -- 172.21.15.107:0/3627893411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179406acf0 con 0x7f17800047d0 2023-12-04T05:46:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f179159a640 1 -- 172.21.15.107:0/3627893411 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178000c040 con 0x7f1780003980 2023-12-04T05:46:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f179159a640 1 -- 172.21.15.107:0/3627893411 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178000c200 con 0x7f1780005190 2023-12-04T05:46:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f1791d9b640 1 --2- 172.21.15.107:0/3627893411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17800047d0 0x7f1780004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f179259c640 1 --2- 172.21.15.107:0/3627893411 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1780003980 0x7f1780003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f1792d9d640 1 --2- 172.21.15.107:0/3627893411 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1780005190 0x7f178000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f1791d9b640 1 -- 172.21.15.107:0/3627893411 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1780005190 msgr2=0x7f178000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f1791d9b640 1 --2- 172.21.15.107:0/3627893411 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1780005190 0x7f178000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f1791d9b640 1 -- 172.21.15.107:0/3627893411 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1780003980 msgr2=0x7f1780003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:10.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f1791d9b640 1 --2- 172.21.15.107:0/3627893411 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1780003980 0x7f1780003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:10.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f1791d9b640 1 -- 172.21.15.107:0/3627893411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1780003530 con 0x7f17800047d0 2023-12-04T05:46:10.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f1792d9d640 1 --2- 172.21.15.107:0/3627893411 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1780005190 0x7f178000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:46:10.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f1792d9d640 1 -- 172.21.15.107:0/3627893411 reap_dead start 2023-12-04T05:46:10.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f1791d9b640 1 --2- 172.21.15.107:0/3627893411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17800047d0 0x7f1780004c50 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f178400a830 tx=0x7f178401d2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8c3b6cfef230905 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:10.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f179159a640 1 -- 172.21.15.107:0/3627893411 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17840a5ea0 con 0x7f17800047d0 2023-12-04T05:46:10.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f179159a640 1 -- 172.21.15.107:0/3627893411 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17840a8070 con 0x7f17800047d0 2023-12-04T05:46:10.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.026+0000 7f179159a640 1 -- 172.21.15.107:0/3627893411 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1784007530 con 0x7f17800047d0 2023-12-04T05:46:10.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f1798dac640 1 -- 172.21.15.107:0/3627893411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17800047d0 msgr2=0x7f1780004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:10.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f1798dac640 1 --2- 172.21.15.107:0/3627893411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17800047d0 0x7f1780004c50 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f178400a830 tx=0x7f178401d2c0 comp rx=0 tx=0).stop 2023-12-04T05:46:10.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f1798dac640 1 -- 172.21.15.107:0/3627893411 shutdown_connections 2023-12-04T05:46:10.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f1798dac640 1 --2- 172.21.15.107:0/3627893411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17800047d0 0x7f1780004c50 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:10.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f1798dac640 1 -- 172.21.15.107:0/3627893411 >> 172.21.15.107:0/3627893411 conn(0x7f17940cc490 msgr2=0x7f17940cc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:10.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f1798dac640 1 -- 172.21.15.107:0/3627893411 shutdown_connections 2023-12-04T05:46:10.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f1798dac640 1 -- 172.21.15.107:0/3627893411 wait complete. 2023-12-04T05:46:10.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f1798dac640 1 Processor -- start 2023-12-04T05:46:10.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f1798dac640 1 -- start start 2023-12-04T05:46:10.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f1798dac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17800047d0 0x7f179414c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:10.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f1798dac640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f179414cb80 0x7f1794146f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:10.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f1798dac640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1794147470 0x7f179415b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:10.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f1798dac640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179415a1b0 con 0x7f17800047d0 2023-12-04T05:46:10.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f1798dac640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17941478f0 con 0x7f179414cb80 2023-12-04T05:46:10.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f1798dac640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17940cbcb0 con 0x7f1794147470 2023-12-04T05:46:10.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f179259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17800047d0 0x7f179414c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:10.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f179259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17800047d0 0x7f179414c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60900/0 (socket says 172.21.15.107:60900) 2023-12-04T05:46:10.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f179259c640 1 -- 172.21.15.107:0/3094689590 learned_addr learned my addr 172.21.15.107:0/3094689590 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:10.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f1791d9b640 1 --2- 172.21.15.107:0/3094689590 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f179414cb80 0x7f1794146f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:10.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f1792d9d640 1 --2- 172.21.15.107:0/3094689590 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1794147470 0x7f179415b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:10.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f179259c640 1 -- 172.21.15.107:0/3094689590 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1794147470 msgr2=0x7f179415b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:10.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f179259c640 1 --2- 172.21.15.107:0/3094689590 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1794147470 0x7f179415b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:10.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f179259c640 1 -- 172.21.15.107:0/3094689590 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f179414cb80 msgr2=0x7f1794146f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:10.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f179259c640 1 --2- 172.21.15.107:0/3094689590 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f179414cb80 0x7f1794146f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:10.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f179259c640 1 -- 172.21.15.107:0/3094689590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17940c2bf0 con 0x7f17800047d0 2023-12-04T05:46:10.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f179259c640 1 --2- 172.21.15.107:0/3094689590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17800047d0 0x7f179414c230 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f177c00e4d0 tx=0x7f177c00e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:10.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f178affd640 1 -- 172.21.15.107:0/3094689590 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c0052e0 con 0x7f17800047d0 2023-12-04T05:46:10.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f178affd640 1 -- 172.21.15.107:0/3094689590 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f177c005440 con 0x7f17800047d0 2023-12-04T05:46:10.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f1798dac640 1 -- 172.21.15.107:0/3094689590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17940c2ed0 con 0x7f17800047d0 2023-12-04T05:46:10.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f178affd640 1 -- 172.21.15.107:0/3094689590 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c003820 con 0x7f17800047d0 2023-12-04T05:46:10.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.030+0000 7f1798dac640 1 -- 172.21.15.107:0/3094689590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17940cb660 con 0x7f17800047d0 2023-12-04T05:46:10.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.034+0000 7f1798dac640 1 -- 172.21.15.107:0/3094689590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1794067ec0 con 0x7f17800047d0 2023-12-04T05:46:10.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.034+0000 7f178affd640 1 -- 172.21.15.107:0/3094689590 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f177c003980 con 0x7f17800047d0 2023-12-04T05:46:10.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.034+0000 7f178affd640 1 --2- 172.21.15.107:0/3094689590 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1774066900 0x7f1774068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:10.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.034+0000 7f178affd640 1 -- 172.21.15.107:0/3094689590 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f177c0840e0 con 0x7f17800047d0 2023-12-04T05:46:10.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.034+0000 7f1791d9b640 1 --2- 172.21.15.107:0/3094689590 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1774066900 0x7f1774068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:10.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.034+0000 7f1791d9b640 1 --2- 172.21.15.107:0/3094689590 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1774066900 0x7f1774068dc0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f17840ad000 tx=0x7f1784009050 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:10.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.038+0000 7f178affd640 1 -- 172.21.15.107:0/3094689590 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f177c0510c0 con 0x7f17800047d0 2023-12-04T05:46:10.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.226+0000 7f1798dac640 1 -- 172.21.15.107:0/3094689590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17941576f0 con 0x7f17800047d0 2023-12-04T05:46:10.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.230+0000 7f178affd640 1 -- 172.21.15.107:0/3094689590 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4154 (secure 0 0 0) 0x7f177c0546e0 con 0x7f17800047d0 2023-12-04T05:46:10.232 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:10.232 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:08.923984+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:10.233 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 49 2023-12-04T05:46:10.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.230+0000 7f1798dac640 1 -- 172.21.15.107:0/3094689590 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1774066900 msgr2=0x7f1774068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:10.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.230+0000 7f1798dac640 1 --2- 172.21.15.107:0/3094689590 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1774066900 0x7f1774068dc0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f17840ad000 tx=0x7f1784009050 comp rx=0 tx=0).stop 2023-12-04T05:46:10.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.230+0000 7f1798dac640 1 -- 172.21.15.107:0/3094689590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17800047d0 msgr2=0x7f179414c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:10.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.230+0000 7f1798dac640 1 --2- 172.21.15.107:0/3094689590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17800047d0 0x7f179414c230 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f177c00e4d0 tx=0x7f177c00e990 comp rx=0 tx=0).stop 2023-12-04T05:46:10.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.230+0000 7f1798dac640 1 -- 172.21.15.107:0/3094689590 shutdown_connections 2023-12-04T05:46:10.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.230+0000 7f1798dac640 1 --2- 172.21.15.107:0/3094689590 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1774066900 0x7f1774068dc0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:10.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.230+0000 7f1798dac640 1 --2- 172.21.15.107:0/3094689590 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1794147470 0x7f179415b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:10.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.230+0000 7f1798dac640 1 --2- 172.21.15.107:0/3094689590 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f179414cb80 0x7f1794146f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:10.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.230+0000 7f1798dac640 1 --2- 172.21.15.107:0/3094689590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17800047d0 0x7f179414c230 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:10.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.230+0000 7f1798dac640 1 -- 172.21.15.107:0/3094689590 >> 172.21.15.107:0/3094689590 conn(0x7f17940cc490 msgr2=0x7f17940c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:10.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.230+0000 7f1798dac640 1 -- 172.21.15.107:0/3094689590 shutdown_connections 2023-12-04T05:46:10.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:10.230+0000 7f1798dac640 1 -- 172.21.15.107:0/3094689590 wait complete. 2023-12-04T05:46:10.251 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:46:12.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:12.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.382+0000 7f4b457d2640 1 Processor -- start 2023-12-04T05:46:12.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.382+0000 7f4b457d2640 1 -- start start 2023-12-04T05:46:12.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.382+0000 7f4b457d2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4b40151180 0x7f4b400c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:12.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.382+0000 7f4b457d2640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4b40070380 0x7f4b40070780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:12.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.382+0000 7f4b457d2640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b400700c0 con 0x7f4b400714f0 2023-12-04T05:46:12.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.382+0000 7f4b457d2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b40156df0 con 0x7f4b40151180 2023-12-04T05:46:12.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.382+0000 7f4b457d2640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b401570d0 con 0x7f4b40070380 2023-12-04T05:46:12.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.382+0000 7f4b3effd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4b40151180 0x7f4b400c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:12.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.382+0000 7f4b3effd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4b40151180 0x7f4b400c79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60912/0 (socket says 172.21.15.107:60912) 2023-12-04T05:46:12.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.382+0000 7f4b3effd640 1 -- 172.21.15.107:0/3770050517 learned_addr learned my addr 172.21.15.107:0/3770050517 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:12.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.382+0000 7f4b3f7fe640 1 --2- 172.21.15.107:0/3770050517 >> v2:172.21.15.195:3300/0 conn(0x7f4b40070380 0x7f4b40070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:12.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.382+0000 7f4b3effd640 1 -- 172.21.15.107:0/3770050517 >> v2:172.21.15.195:3300/0 conn(0x7f4b40070380 msgr2=0x7f4b40070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:12.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.382+0000 7f4b3effd640 1 --2- 172.21.15.107:0/3770050517 >> v2:172.21.15.195:3300/0 conn(0x7f4b40070380 0x7f4b40070780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:12.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.382+0000 7f4b3effd640 1 -- 172.21.15.107:0/3770050517 >> v1:172.21.15.195:6789/0 conn(0x7f4b400714f0 legacy=0x7f4b40071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:12.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.382+0000 7f4b3effd640 1 -- 172.21.15.107:0/3770050517 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b401573b0 con 0x7f4b40151180 2023-12-04T05:46:12.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.382+0000 7f4b3f7fe640 1 --2- 172.21.15.107:0/3770050517 >> v2:172.21.15.195:3300/0 conn(0x7f4b40070380 0x7f4b40070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:46:12.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b3effd640 1 --2- 172.21.15.107:0/3770050517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b40151180 0x7f4b400c79e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4b4006f680 tx=0x7f4b2801ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c86dc9641de2c83 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:12.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b3e7fc640 1 -- 172.21.15.107:0/3770050517 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b280a89b0 con 0x7f4b40151180 2023-12-04T05:46:12.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b3e7fc640 1 -- 172.21.15.107:0/3770050517 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b280a8b10 con 0x7f4b40151180 2023-12-04T05:46:12.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b3e7fc640 1 -- 172.21.15.107:0/3770050517 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b280a8ea0 con 0x7f4b40151180 2023-12-04T05:46:12.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b457d2640 1 -- 172.21.15.107:0/3770050517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b40151180 msgr2=0x7f4b400c79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:12.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b457d2640 1 --2- 172.21.15.107:0/3770050517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b40151180 0x7f4b400c79e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4b4006f680 tx=0x7f4b2801ecb0 comp rx=0 tx=0).stop 2023-12-04T05:46:12.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b457d2640 1 -- 172.21.15.107:0/3770050517 shutdown_connections 2023-12-04T05:46:12.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b457d2640 1 --2- 172.21.15.107:0/3770050517 >> v2:172.21.15.195:3300/0 conn(0x7f4b40070380 0x7f4b40070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:12.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b457d2640 1 --2- 172.21.15.107:0/3770050517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b40151180 0x7f4b400c79e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:12.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b457d2640 1 -- 172.21.15.107:0/3770050517 >> 172.21.15.107:0/3770050517 conn(0x7f4b400c7ff0 msgr2=0x7f4b400c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:12.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b457d2640 1 -- 172.21.15.107:0/3770050517 shutdown_connections 2023-12-04T05:46:12.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b457d2640 1 -- 172.21.15.107:0/3770050517 wait complete. 2023-12-04T05:46:12.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b457d2640 1 Processor -- start 2023-12-04T05:46:12.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b457d2640 1 -- start start 2023-12-04T05:46:12.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b457d2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b40070380 0x7f4b401efdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:12.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b457d2640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4b400714f0 0x7f4b401ea000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:12.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b457d2640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4b40151180 0x7f4b40142170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:12.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b457d2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b401ea540 con 0x7f4b40070380 2023-12-04T05:46:12.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b457d2640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b400c5eb0 con 0x7f4b40151180 2023-12-04T05:46:12.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b457d2640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b400c6190 con 0x7f4b400714f0 2023-12-04T05:46:12.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b3effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b40070380 0x7f4b401efdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:12.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b3effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b40070380 0x7f4b401efdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60914/0 (socket says 172.21.15.107:60914) 2023-12-04T05:46:12.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b3effd640 1 -- 172.21.15.107:0/1113981241 learned_addr learned my addr 172.21.15.107:0/1113981241 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:12.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b3effd640 1 -- 172.21.15.107:0/1113981241 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4b400714f0 msgr2=0x7f4b401ea000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:12.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b37fff640 1 --2- 172.21.15.107:0/1113981241 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4b400714f0 0x7f4b401ea000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:12.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b3f7fe640 1 --2- 172.21.15.107:0/1113981241 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4b40151180 0x7f4b40142170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:12.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b3effd640 1 --2- 172.21.15.107:0/1113981241 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4b400714f0 0x7f4b401ea000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:12.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b3effd640 1 -- 172.21.15.107:0/1113981241 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4b40151180 msgr2=0x7f4b40142170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:12.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b3effd640 1 --2- 172.21.15.107:0/1113981241 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4b40151180 0x7f4b40142170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:12.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b3effd640 1 -- 172.21.15.107:0/1113981241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b280094d0 con 0x7f4b40070380 2023-12-04T05:46:12.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.386+0000 7f4b3effd640 1 --2- 172.21.15.107:0/1113981241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b40070380 0x7f4b401efdd0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4b280b3040 tx=0x7f4b2801ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:12.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.390+0000 7f4b377fe640 1 -- 172.21.15.107:0/1113981241 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b280b0e80 con 0x7f4b40070380 2023-12-04T05:46:12.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.390+0000 7f4b377fe640 1 -- 172.21.15.107:0/1113981241 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b280bb480 con 0x7f4b40070380 2023-12-04T05:46:12.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.390+0000 7f4b457d2640 1 -- 172.21.15.107:0/1113981241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b400c6470 con 0x7f4b40070380 2023-12-04T05:46:12.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.390+0000 7f4b377fe640 1 -- 172.21.15.107:0/1113981241 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b280bb830 con 0x7f4b40070380 2023-12-04T05:46:12.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.390+0000 7f4b457d2640 1 -- 172.21.15.107:0/1113981241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b401e9800 con 0x7f4b40070380 2023-12-04T05:46:12.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.390+0000 7f4b457d2640 1 -- 172.21.15.107:0/1113981241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b400c6660 con 0x7f4b40070380 2023-12-04T05:46:12.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.390+0000 7f4b377fe640 1 -- 172.21.15.107:0/1113981241 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4b280bb990 con 0x7f4b40070380 2023-12-04T05:46:12.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.390+0000 7f4b377fe640 1 --2- 172.21.15.107:0/1113981241 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4b20066900 0x7f4b20068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:12.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.390+0000 7f4b377fe640 1 -- 172.21.15.107:0/1113981241 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f4b28122cc0 con 0x7f4b40070380 2023-12-04T05:46:12.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.390+0000 7f4b37fff640 1 --2- 172.21.15.107:0/1113981241 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4b20066900 0x7f4b20068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:12.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.390+0000 7f4b37fff640 1 --2- 172.21.15.107:0/1113981241 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4b20066900 0x7f4b20068dc0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4b30007fd0 tx=0x7f4b30009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:12.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.394+0000 7f4b377fe640 1 -- 172.21.15.107:0/1113981241 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4b280efe50 con 0x7f4b40070380 2023-12-04T05:46:12.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.578+0000 7f4b457d2640 1 -- 172.21.15.107:0/1113981241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b400c6660 con 0x7f4b40070380 2023-12-04T05:46:12.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.578+0000 7f4b377fe640 1 -- 172.21.15.107:0/1113981241 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f4b280b5090 con 0x7f4b40070380 2023-12-04T05:46:12.583 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:12.583 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:12.583 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:46:12.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.582+0000 7f4b457d2640 1 -- 172.21.15.107:0/1113981241 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4b20066900 msgr2=0x7f4b20068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:12.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.582+0000 7f4b457d2640 1 --2- 172.21.15.107:0/1113981241 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4b20066900 0x7f4b20068dc0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4b30007fd0 tx=0x7f4b30009040 comp rx=0 tx=0).stop 2023-12-04T05:46:12.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.582+0000 7f4b457d2640 1 -- 172.21.15.107:0/1113981241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b40070380 msgr2=0x7f4b401efdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.582+0000 7f4b457d2640 1 --2- 172.21.15.107:0/1113981241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b40070380 0x7f4b401efdd0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4b280b3040 tx=0x7f4b2801ba50 comp rx=0 tx=0).stop 2023-12-04T05:46:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.582+0000 7f4b457d2640 1 -- 172.21.15.107:0/1113981241 shutdown_connections 2023-12-04T05:46:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.582+0000 7f4b457d2640 1 --2- 172.21.15.107:0/1113981241 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4b20066900 0x7f4b20068dc0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.582+0000 7f4b457d2640 1 --2- 172.21.15.107:0/1113981241 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4b40151180 0x7f4b40142170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.582+0000 7f4b457d2640 1 --2- 172.21.15.107:0/1113981241 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4b400714f0 0x7f4b401ea000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.582+0000 7f4b457d2640 1 --2- 172.21.15.107:0/1113981241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b40070380 0x7f4b401efdd0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.582+0000 7f4b457d2640 1 -- 172.21.15.107:0/1113981241 >> 172.21.15.107:0/1113981241 conn(0x7f4b400c7ff0 msgr2=0x7f4b40072090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.582+0000 7f4b457d2640 1 -- 172.21.15.107:0/1113981241 shutdown_connections 2023-12-04T05:46:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:12.582+0000 7f4b457d2640 1 -- 172.21.15.107:0/1113981241 wait complete. 2023-12-04T05:46:12.600 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:46:14.601 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:14.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.734+0000 7f3fdc6bd640 1 Processor -- start 2023-12-04T05:46:14.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.734+0000 7f3fdc6bd640 1 -- start start 2023-12-04T05:46:14.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.734+0000 7f3fdc6bd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3fd4072110 0x7f3fd4158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:14.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.734+0000 7f3fdc6bd640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3fd40724f0 0x7f3fd40c7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:14.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.734+0000 7f3fdc6bd640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd4158da0 con 0x7f3fd40c56f0 2023-12-04T05:46:14.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.734+0000 7f3fdc6bd640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fd41583d0 con 0x7f3fd4072110 2023-12-04T05:46:14.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.734+0000 7f3fdc6bd640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd40c6170 con 0x7f3fd40724f0 2023-12-04T05:46:14.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.734+0000 7f3fda459640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3fd4072110 0x7f3fd4158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:14.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.734+0000 7f3fdac5a640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3fd40c56f0 0x7f3fd4140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55194/0 (socket says 172.21.15.107:55194) 2023-12-04T05:46:14.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.734+0000 7f3fdac5a640 1 -- 172.21.15.107:0/3006912553 learned_addr learned my addr 172.21.15.107:0/3006912553 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:46:14.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.734+0000 7f3fd9c58640 1 --2- 172.21.15.107:0/3006912553 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3fd40724f0 0x7f3fd40c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:14.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.734+0000 7f3fda459640 1 -- 172.21.15.107:0/3006912553 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3fd40724f0 msgr2=0x7f3fd40c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:14.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.734+0000 7f3fda459640 1 --2- 172.21.15.107:0/3006912553 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3fd40724f0 0x7f3fd40c7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:14.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.734+0000 7f3fda459640 1 -- 172.21.15.107:0/3006912553 >> v1:172.21.15.107:6789/0 conn(0x7f3fd40c56f0 legacy=0x7f3fd4140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:14.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.734+0000 7f3fd9457640 1 -- 172.21.15.107:0/3006912553 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2475713101 0 0) 0x7f3fcc0014a0 con 0x7f3fd40c56f0 2023-12-04T05:46:14.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.734+0000 7f3fda459640 1 -- 172.21.15.107:0/3006912553 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd4157da0 con 0x7f3fd4072110 2023-12-04T05:46:14.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.734+0000 7f3fda459640 1 --2- 172.21.15.107:0/3006912553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fd4072110 0x7f3fd4158860 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f3fd00097c0 tx=0x7f3fd001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed28cae5eda21de3 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:14.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fd9457640 1 -- 172.21.15.107:0/3006912553 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd00a8970 con 0x7f3fd4072110 2023-12-04T05:46:14.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fd9457640 1 -- 172.21.15.107:0/3006912553 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd00a8ad0 con 0x7f3fd4072110 2023-12-04T05:46:14.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fd9457640 1 -- 172.21.15.107:0/3006912553 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd00a8e60 con 0x7f3fd4072110 2023-12-04T05:46:14.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fdc6bd640 1 -- 172.21.15.107:0/3006912553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fd4072110 msgr2=0x7f3fd4158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:14.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fdc6bd640 1 --2- 172.21.15.107:0/3006912553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fd4072110 0x7f3fd4158860 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f3fd00097c0 tx=0x7f3fd001ec70 comp rx=0 tx=0).stop 2023-12-04T05:46:14.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fdc6bd640 1 -- 172.21.15.107:0/3006912553 shutdown_connections 2023-12-04T05:46:14.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fdc6bd640 1 --2- 172.21.15.107:0/3006912553 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3fd40724f0 0x7f3fd40c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:14.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fdc6bd640 1 --2- 172.21.15.107:0/3006912553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fd4072110 0x7f3fd4158860 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:14.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fdc6bd640 1 -- 172.21.15.107:0/3006912553 >> 172.21.15.107:0/3006912553 conn(0x7f3fd40c7ff0 msgr2=0x7f3fd40c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:14.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fdc6bd640 1 -- 172.21.15.107:0/3006912553 shutdown_connections 2023-12-04T05:46:14.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fdc6bd640 1 -- 172.21.15.107:0/3006912553 wait complete. 2023-12-04T05:46:14.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fdc6bd640 1 Processor -- start 2023-12-04T05:46:14.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fdc6bd640 1 -- start start 2023-12-04T05:46:14.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fdc6bd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fd4072110 0x7f3fd414d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:14.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fdc6bd640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3fd40724f0 0x7f3fd4158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:14.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fdc6bd640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3fd40c56f0 0x7f3fd4154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:14.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fdc6bd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd415a420 con 0x7f3fd4072110 2023-12-04T05:46:14.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fdc6bd640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd4152690 con 0x7f3fd40724f0 2023-12-04T05:46:14.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fdc6bd640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd414d780 con 0x7f3fd40c56f0 2023-12-04T05:46:14.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fda459640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fd4072110 0x7f3fd414d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:14.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fda459640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fd4072110 0x7f3fd414d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60946/0 (socket says 172.21.15.107:60946) 2023-12-04T05:46:14.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fda459640 1 -- 172.21.15.107:0/1859295014 learned_addr learned my addr 172.21.15.107:0/1859295014 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:14.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fd9c58640 1 --2- 172.21.15.107:0/1859295014 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3fd40724f0 0x7f3fd4158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:14.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fdac5a640 1 --2- 172.21.15.107:0/1859295014 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3fd40c56f0 0x7f3fd4154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:14.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fda459640 1 -- 172.21.15.107:0/1859295014 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3fd40c56f0 msgr2=0x7f3fd4154c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:14.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fda459640 1 --2- 172.21.15.107:0/1859295014 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3fd40c56f0 0x7f3fd4154c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:14.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fda459640 1 -- 172.21.15.107:0/1859295014 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3fd40724f0 msgr2=0x7f3fd4158f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:14.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fda459640 1 --2- 172.21.15.107:0/1859295014 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3fd40724f0 0x7f3fd4158f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:14.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fda459640 1 -- 172.21.15.107:0/1859295014 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd0009420 con 0x7f3fd4072110 2023-12-04T05:46:14.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fda459640 1 --2- 172.21.15.107:0/1859295014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fd4072110 0x7f3fd414d240 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f3fd00097c0 tx=0x7f3fd001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:14.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fcaffd640 1 -- 172.21.15.107:0/1859295014 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd00b9630 con 0x7f3fd4072110 2023-12-04T05:46:14.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fcaffd640 1 -- 172.21.15.107:0/1859295014 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd00b97b0 con 0x7f3fd4072110 2023-12-04T05:46:14.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.738+0000 7f3fdc6bd640 1 -- 172.21.15.107:0/1859295014 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fd41528e0 con 0x7f3fd4072110 2023-12-04T05:46:14.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.742+0000 7f3fcaffd640 1 -- 172.21.15.107:0/1859295014 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd00b9b60 con 0x7f3fd4072110 2023-12-04T05:46:14.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.742+0000 7f3fdc6bd640 1 -- 172.21.15.107:0/1859295014 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fd4152ea0 con 0x7f3fd4072110 2023-12-04T05:46:14.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.742+0000 7f3fdc6bd640 1 -- 172.21.15.107:0/1859295014 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd4067870 con 0x7f3fd4072110 2023-12-04T05:46:14.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.742+0000 7f3fcaffd640 1 -- 172.21.15.107:0/1859295014 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f3fd00b7040 con 0x7f3fd4072110 2023-12-04T05:46:14.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.742+0000 7f3fcaffd640 1 --2- 172.21.15.107:0/1859295014 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3fb80668d0 0x7f3fb8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:14.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.742+0000 7f3fcaffd640 1 -- 172.21.15.107:0/1859295014 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f3fd0122530 con 0x7f3fd4072110 2023-12-04T05:46:14.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.742+0000 7f3fd9c58640 1 --2- 172.21.15.107:0/1859295014 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3fb80668d0 0x7f3fb8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:14.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.742+0000 7f3fd9c58640 1 --2- 172.21.15.107:0/1859295014 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3fb80668d0 0x7f3fb8068d90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4152350 tx=0x7f3fc4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:14.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.746+0000 7f3fcaffd640 1 -- 172.21.15.107:0/1859295014 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3fd00ef6c0 con 0x7f3fd4072110 2023-12-04T05:46:14.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~33s 2023-12-04T05:46:14.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~147s 2023-12-04T05:46:14.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~188s 2023-12-04T05:46:14.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.934+0000 7f3fdc6bd640 1 -- 172.21.15.107:0/1859295014 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fd4153bb0 con 0x7f3fd4072110 2023-12-04T05:46:14.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.934+0000 7f3fcaffd640 1 -- 172.21.15.107:0/1859295014 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f3fd00b5090 con 0x7f3fd4072110 2023-12-04T05:46:14.941 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:14.941 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:14.942 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:46:14.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.938+0000 7f3fdc6bd640 1 -- 172.21.15.107:0/1859295014 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3fb80668d0 msgr2=0x7f3fb8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:14.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.938+0000 7f3fdc6bd640 1 --2- 172.21.15.107:0/1859295014 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3fb80668d0 0x7f3fb8068d90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4152350 tx=0x7f3fc4008040 comp rx=0 tx=0).stop 2023-12-04T05:46:14.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.938+0000 7f3fdc6bd640 1 -- 172.21.15.107:0/1859295014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fd4072110 msgr2=0x7f3fd414d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:14.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.938+0000 7f3fdc6bd640 1 --2- 172.21.15.107:0/1859295014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fd4072110 0x7f3fd414d240 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f3fd00097c0 tx=0x7f3fd001ba10 comp rx=0 tx=0).stop 2023-12-04T05:46:14.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.938+0000 7f3fdc6bd640 1 -- 172.21.15.107:0/1859295014 shutdown_connections 2023-12-04T05:46:14.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.938+0000 7f3fdc6bd640 1 --2- 172.21.15.107:0/1859295014 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3fb80668d0 0x7f3fb8068d90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:14.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.938+0000 7f3fdc6bd640 1 --2- 172.21.15.107:0/1859295014 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3fd40c56f0 0x7f3fd4154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:14.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.938+0000 7f3fdc6bd640 1 --2- 172.21.15.107:0/1859295014 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3fd40724f0 0x7f3fd4158f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:14.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.938+0000 7f3fdc6bd640 1 --2- 172.21.15.107:0/1859295014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fd4072110 0x7f3fd414d240 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:14.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.938+0000 7f3fdc6bd640 1 -- 172.21.15.107:0/1859295014 >> 172.21.15.107:0/1859295014 conn(0x7f3fd40c7ff0 msgr2=0x7f3fd40c6340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:14.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.938+0000 7f3fdc6bd640 1 -- 172.21.15.107:0/1859295014 shutdown_connections 2023-12-04T05:46:14.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:14.938+0000 7f3fdc6bd640 1 -- 172.21.15.107:0/1859295014 wait complete. 2023-12-04T05:46:14.957 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:46:16.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:17.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.086+0000 7fce8380a640 1 Processor -- start 2023-12-04T05:46:17.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.086+0000 7fce8380a640 1 -- start start 2023-12-04T05:46:17.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.086+0000 7fce8380a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fce7c0c46a0 0x7fce7c0c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:17.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.086+0000 7fce8380a640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fce7c0c4a80 0x7fce7c0c7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:17.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.086+0000 7fce8380a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce7c06e660 con 0x7fce7c157180 2023-12-04T05:46:17.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.086+0000 7fce8380a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce7c156490 con 0x7fce7c0c46a0 2023-12-04T05:46:17.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.086+0000 7fce8380a640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce7c156cf0 con 0x7fce7c0c4a80 2023-12-04T05:46:17.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.086+0000 7fce815a6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fce7c0c46a0 0x7fce7c0c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:17.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.086+0000 7fce80da5640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fce7c157180 0x7fce7c0c5ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55200/0 (socket says 172.21.15.107:55200) 2023-12-04T05:46:17.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.086+0000 7fce80da5640 1 -- 172.21.15.107:0/4011152773 learned_addr learned my addr 172.21.15.107:0/4011152773 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:46:17.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.086+0000 7fce81da7640 1 --2- 172.21.15.107:0/4011152773 >> v2:172.21.15.195:3300/0 conn(0x7fce7c0c4a80 0x7fce7c0c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:17.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.086+0000 7fce815a6640 1 -- 172.21.15.107:0/4011152773 >> v2:172.21.15.195:3300/0 conn(0x7fce7c0c4a80 msgr2=0x7fce7c0c7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:17.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.086+0000 7fce815a6640 1 --2- 172.21.15.107:0/4011152773 >> v2:172.21.15.195:3300/0 conn(0x7fce7c0c4a80 0x7fce7c0c7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:17.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.086+0000 7fce815a6640 1 -- 172.21.15.107:0/4011152773 >> v1:172.21.15.107:6789/0 conn(0x7fce7c157180 legacy=0x7fce7c0c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:17.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.086+0000 7fce815a6640 1 -- 172.21.15.107:0/4011152773 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce7c06e0a0 con 0x7fce7c0c46a0 2023-12-04T05:46:17.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.086+0000 7fce73fff640 1 -- 172.21.15.107:0/4011152773 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660866196 0 0) 0x7fce6c001120 con 0x7fce7c157180 2023-12-04T05:46:17.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.090+0000 7fce815a6640 1 --2- 172.21.15.107:0/4011152773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce7c0c46a0 0x7fce7c0c8b10 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fce7c0708d0 tx=0x7fce7801d830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c8938af37adf938 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:17.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.090+0000 7fce73fff640 1 -- 172.21.15.107:0/4011152773 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce780a67d0 con 0x7fce7c0c46a0 2023-12-04T05:46:17.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.090+0000 7fce73fff640 1 -- 172.21.15.107:0/4011152773 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce780a6930 con 0x7fce7c0c46a0 2023-12-04T05:46:17.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.090+0000 7fce73fff640 1 -- 172.21.15.107:0/4011152773 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce780a6cc0 con 0x7fce7c0c46a0 2023-12-04T05:46:17.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.090+0000 7fce8380a640 1 -- 172.21.15.107:0/4011152773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce7c0c46a0 msgr2=0x7fce7c0c8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:17.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.090+0000 7fce8380a640 1 --2- 172.21.15.107:0/4011152773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce7c0c46a0 0x7fce7c0c8b10 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fce7c0708d0 tx=0x7fce7801d830 comp rx=0 tx=0).stop 2023-12-04T05:46:17.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.090+0000 7fce8380a640 1 -- 172.21.15.107:0/4011152773 shutdown_connections 2023-12-04T05:46:17.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.090+0000 7fce8380a640 1 --2- 172.21.15.107:0/4011152773 >> v2:172.21.15.195:3300/0 conn(0x7fce7c0c4a80 0x7fce7c0c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:17.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.090+0000 7fce8380a640 1 --2- 172.21.15.107:0/4011152773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce7c0c46a0 0x7fce7c0c8b10 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:17.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.090+0000 7fce8380a640 1 -- 172.21.15.107:0/4011152773 >> 172.21.15.107:0/4011152773 conn(0x7fce7c0c7ff0 msgr2=0x7fce7c0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:17.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.090+0000 7fce8380a640 1 -- 172.21.15.107:0/4011152773 shutdown_connections 2023-12-04T05:46:17.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.090+0000 7fce8380a640 1 -- 172.21.15.107:0/4011152773 wait complete. 2023-12-04T05:46:17.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.090+0000 7fce8380a640 1 Processor -- start 2023-12-04T05:46:17.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.090+0000 7fce8380a640 1 -- start start 2023-12-04T05:46:17.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.090+0000 7fce8380a640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fce7c0c46a0 0x7fce7c146300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:17.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.090+0000 7fce8380a640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce7c0c4a80 0x7fce7c141ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:17.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.090+0000 7fce8380a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce7c157180 0x7fce7c1f4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:17.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.090+0000 7fce8380a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7c149570 con 0x7fce7c157180 2023-12-04T05:46:17.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.090+0000 7fce8380a640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7c149850 con 0x7fce7c0c4a80 2023-12-04T05:46:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.090+0000 7fce8380a640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce7c149b30 con 0x7fce7c0c46a0 2023-12-04T05:46:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.090+0000 7fce81da7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce7c157180 0x7fce7c1f4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.094+0000 7fce81da7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce7c157180 0x7fce7c1f4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60962/0 (socket says 172.21.15.107:60962) 2023-12-04T05:46:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.094+0000 7fce81da7640 1 -- 172.21.15.107:0/4042711084 learned_addr learned my addr 172.21.15.107:0/4042711084 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.094+0000 7fce815a6640 1 --2- 172.21.15.107:0/4042711084 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fce7c0c46a0 0x7fce7c146300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.094+0000 7fce80da5640 1 --2- 172.21.15.107:0/4042711084 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce7c0c4a80 0x7fce7c141ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.094+0000 7fce81da7640 1 -- 172.21.15.107:0/4042711084 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fce7c0c46a0 msgr2=0x7fce7c146300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.094+0000 7fce81da7640 1 --2- 172.21.15.107:0/4042711084 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fce7c0c46a0 0x7fce7c146300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.094+0000 7fce81da7640 1 -- 172.21.15.107:0/4042711084 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce7c0c4a80 msgr2=0x7fce7c141ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.094+0000 7fce81da7640 1 --2- 172.21.15.107:0/4042711084 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce7c0c4a80 0x7fce7c141ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.094+0000 7fce81da7640 1 -- 172.21.15.107:0/4042711084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce7800a5c0 con 0x7fce7c157180 2023-12-04T05:46:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.094+0000 7fce80da5640 1 --2- 172.21.15.107:0/4042711084 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce7c0c4a80 0x7fce7c141ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:46:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.094+0000 7fce81da7640 1 --2- 172.21.15.107:0/4042711084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce7c157180 0x7fce7c1f4170 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fce74007e40 tx=0x7fce7400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.094+0000 7fce71ffb640 1 -- 172.21.15.107:0/4042711084 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce74014620 con 0x7fce7c157180 2023-12-04T05:46:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.094+0000 7fce71ffb640 1 -- 172.21.15.107:0/4042711084 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7400ee90 con 0x7fce7c157180 2023-12-04T05:46:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.094+0000 7fce8380a640 1 -- 172.21.15.107:0/4042711084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce7c149e10 con 0x7fce7c157180 2023-12-04T05:46:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.094+0000 7fce71ffb640 1 -- 172.21.15.107:0/4042711084 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce740149e0 con 0x7fce7c157180 2023-12-04T05:46:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.094+0000 7fce8380a640 1 -- 172.21.15.107:0/4042711084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce7c1f3c50 con 0x7fce7c157180 2023-12-04T05:46:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.094+0000 7fce8380a640 1 -- 172.21.15.107:0/4042711084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce7c067870 con 0x7fce7c157180 2023-12-04T05:46:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.094+0000 7fce71ffb640 1 -- 172.21.15.107:0/4042711084 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fce74014b40 con 0x7fce7c157180 2023-12-04T05:46:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.094+0000 7fce71ffb640 1 --2- 172.21.15.107:0/4042711084 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fce600668d0 0x7fce60068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:17.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.094+0000 7fce815a6640 1 --2- 172.21.15.107:0/4042711084 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fce600668d0 0x7fce60068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:17.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.094+0000 7fce71ffb640 1 -- 172.21.15.107:0/4042711084 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fce74085040 con 0x7fce7c157180 2023-12-04T05:46:17.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.094+0000 7fce815a6640 1 --2- 172.21.15.107:0/4042711084 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fce600668d0 0x7fce60068d90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fce780b0000 tx=0x7fce78008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:17.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.098+0000 7fce71ffb640 1 -- 172.21.15.107:0/4042711084 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fce74051fa0 con 0x7fce7c157180 2023-12-04T05:46:17.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.286+0000 7fce8380a640 1 -- 172.21.15.107:0/4042711084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce7c1f3e10 con 0x7fce7c157180 2023-12-04T05:46:17.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.290+0000 7fce71ffb640 1 -- 172.21.15.107:0/4042711084 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7fce740555c0 con 0x7fce7c157180 2023-12-04T05:46:17.291 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:17.291 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:17.291 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:46:17.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.290+0000 7fce8380a640 1 -- 172.21.15.107:0/4042711084 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fce600668d0 msgr2=0x7fce60068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:17.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.290+0000 7fce8380a640 1 --2- 172.21.15.107:0/4042711084 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fce600668d0 0x7fce60068d90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fce780b0000 tx=0x7fce78008040 comp rx=0 tx=0).stop 2023-12-04T05:46:17.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.290+0000 7fce8380a640 1 -- 172.21.15.107:0/4042711084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce7c157180 msgr2=0x7fce7c1f4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:17.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.290+0000 7fce8380a640 1 --2- 172.21.15.107:0/4042711084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce7c157180 0x7fce7c1f4170 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fce74007e40 tx=0x7fce7400e4b0 comp rx=0 tx=0).stop 2023-12-04T05:46:17.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.290+0000 7fce8380a640 1 -- 172.21.15.107:0/4042711084 shutdown_connections 2023-12-04T05:46:17.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.290+0000 7fce8380a640 1 --2- 172.21.15.107:0/4042711084 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fce600668d0 0x7fce60068d90 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:17.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.290+0000 7fce8380a640 1 --2- 172.21.15.107:0/4042711084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce7c157180 0x7fce7c1f4170 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:17.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.290+0000 7fce8380a640 1 --2- 172.21.15.107:0/4042711084 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fce7c0c4a80 0x7fce7c141ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:17.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.290+0000 7fce8380a640 1 --2- 172.21.15.107:0/4042711084 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fce7c0c46a0 0x7fce7c146300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:17.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.290+0000 7fce8380a640 1 -- 172.21.15.107:0/4042711084 >> 172.21.15.107:0/4042711084 conn(0x7fce7c0c7ff0 msgr2=0x7fce7c153d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:17.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.290+0000 7fce8380a640 1 -- 172.21.15.107:0/4042711084 shutdown_connections 2023-12-04T05:46:17.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:17.290+0000 7fce8380a640 1 -- 172.21.15.107:0/4042711084 wait complete. 2023-12-04T05:46:17.311 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 50, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-12-04T05:39:59.032019+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_4288": { "addr": "172.21.15.195:6837/3412865477", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6835", "nonce": 3412865477, "type": "v2" }, { "addr": "172.21.15.195:6837", "nonce": 3412865477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_5190": { "addr": "172.21.15.107:6836/2928623624", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 2928623624, "type": "v2" }, { "addr": "172.21.15.107:6836", "nonce": 2928623624, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5190, "incarnation": 45, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:rejoin", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-12-04T05:46:11.040247+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2, 3, 4 ], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 5190 } } } ], "standbys": [ { "addr": "172.21.15.107:6837/2752497349", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6835", "nonce": 2752497349, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 2752497349, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 49, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5292, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-12-04T05:46:19.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:19.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.438+0000 7f4d78076640 1 Processor -- start 2023-12-04T05:46:19.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.438+0000 7f4d78076640 1 -- start start 2023-12-04T05:46:19.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.442+0000 7f4d78076640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4d70157180 0x7f4d700c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:19.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.442+0000 7f4d78076640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4d700c4a80 0x7f4d700c7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:19.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.442+0000 7f4d78076640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d7006e660 con 0x7f4d700c46a0 2023-12-04T05:46:19.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.442+0000 7f4d78076640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d70156490 con 0x7f4d70157180 2023-12-04T05:46:19.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.442+0000 7f4d78076640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d70156cf0 con 0x7f4d700c4a80 2023-12-04T05:46:19.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.442+0000 7f4d75e12640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4d700c46a0 0x7f4d700c5ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37974/0 (socket says 172.21.15.107:37974) 2023-12-04T05:46:19.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.442+0000 7f4d75e12640 1 -- 172.21.15.107:0/2167087276 learned_addr learned my addr 172.21.15.107:0/2167087276 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:46:19.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.442+0000 7f4d75611640 1 --2- 172.21.15.107:0/2167087276 >> v2:172.21.15.107:3300/0 conn(0x7f4d70157180 0x7f4d700c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:19.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.442+0000 7f4d76613640 1 --2- 172.21.15.107:0/2167087276 >> v2:172.21.15.195:3300/0 conn(0x7f4d700c4a80 0x7f4d700c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:19.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.442+0000 7f4d74e10640 1 -- 172.21.15.107:0/2167087276 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689217377 0 0) 0x7f4d6c001120 con 0x7f4d700c46a0 2023-12-04T05:46:19.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.442+0000 7f4d74e10640 1 -- 172.21.15.107:0/2167087276 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d5c003530 con 0x7f4d700c46a0 2023-12-04T05:46:19.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.442+0000 7f4d75611640 1 -- 172.21.15.107:0/2167087276 >> v2:172.21.15.195:3300/0 conn(0x7f4d700c4a80 msgr2=0x7f4d700c7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:19.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.442+0000 7f4d75611640 1 --2- 172.21.15.107:0/2167087276 >> v2:172.21.15.195:3300/0 conn(0x7f4d700c4a80 0x7f4d700c7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:19.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.442+0000 7f4d75611640 1 -- 172.21.15.107:0/2167087276 >> v1:172.21.15.107:6789/0 conn(0x7f4d700c46a0 legacy=0x7f4d700c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:19.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.442+0000 7f4d75611640 1 -- 172.21.15.107:0/2167087276 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d7006e0a0 con 0x7f4d70157180 2023-12-04T05:46:19.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.442+0000 7f4d75611640 1 --2- 172.21.15.107:0/2167087276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d70157180 0x7f4d700c8b10 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f4d700708d0 tx=0x7f4d6001d830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34b0243add756366 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:19.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.442+0000 7f4d74e10640 1 -- 172.21.15.107:0/2167087276 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d600a67d0 con 0x7f4d70157180 2023-12-04T05:46:19.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.442+0000 7f4d74e10640 1 -- 172.21.15.107:0/2167087276 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d600a6930 con 0x7f4d70157180 2023-12-04T05:46:19.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.442+0000 7f4d74e10640 1 -- 172.21.15.107:0/2167087276 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d600a6cc0 con 0x7f4d70157180 2023-12-04T05:46:19.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d78076640 1 -- 172.21.15.107:0/2167087276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d70157180 msgr2=0x7f4d700c8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:19.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d78076640 1 --2- 172.21.15.107:0/2167087276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d70157180 0x7f4d700c8b10 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f4d700708d0 tx=0x7f4d6001d830 comp rx=0 tx=0).stop 2023-12-04T05:46:19.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d78076640 1 -- 172.21.15.107:0/2167087276 shutdown_connections 2023-12-04T05:46:19.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d78076640 1 --2- 172.21.15.107:0/2167087276 >> v2:172.21.15.195:3300/0 conn(0x7f4d700c4a80 0x7f4d700c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:19.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d78076640 1 --2- 172.21.15.107:0/2167087276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d70157180 0x7f4d700c8b10 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:19.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d78076640 1 -- 172.21.15.107:0/2167087276 >> 172.21.15.107:0/2167087276 conn(0x7f4d700c7ff0 msgr2=0x7f4d700c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:19.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d78076640 1 -- 172.21.15.107:0/2167087276 shutdown_connections 2023-12-04T05:46:19.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d78076640 1 -- 172.21.15.107:0/2167087276 wait complete. 2023-12-04T05:46:19.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d78076640 1 Processor -- start 2023-12-04T05:46:19.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d78076640 1 -- start start 2023-12-04T05:46:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d78076640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d700c46a0 0x7f4d700c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d78076640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d700c4a80 0x7f4d7014b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d78076640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d70157180 0x7f4d7014b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d78076640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d700ca530 con 0x7f4d700c46a0 2023-12-04T05:46:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d78076640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d701f3020 con 0x7f4d70157180 2023-12-04T05:46:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d78076640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d701f3300 con 0x7f4d700c4a80 2023-12-04T05:46:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d75e12640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d700c46a0 0x7f4d700c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d75e12640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d700c46a0 0x7f4d700c6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53310/0 (socket says 172.21.15.107:53310) 2023-12-04T05:46:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d75e12640 1 -- 172.21.15.107:0/156029863 learned_addr learned my addr 172.21.15.107:0/156029863 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d75611640 1 --2- 172.21.15.107:0/156029863 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d700c4a80 0x7f4d7014b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:19.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d76613640 1 --2- 172.21.15.107:0/156029863 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d70157180 0x7f4d7014b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:19.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d75e12640 1 -- 172.21.15.107:0/156029863 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d700c4a80 msgr2=0x7f4d7014b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:19.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d75e12640 1 --2- 172.21.15.107:0/156029863 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d700c4a80 0x7f4d7014b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:19.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d75e12640 1 -- 172.21.15.107:0/156029863 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d70157180 msgr2=0x7f4d7014b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:19.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d75e12640 1 --2- 172.21.15.107:0/156029863 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d70157180 0x7f4d7014b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:19.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d75e12640 1 -- 172.21.15.107:0/156029863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6000a5c0 con 0x7f4d700c46a0 2023-12-04T05:46:19.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d75611640 1 --2- 172.21.15.107:0/156029863 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d700c4a80 0x7f4d7014b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:46:19.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d76613640 1 --2- 172.21.15.107:0/156029863 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d70157180 0x7f4d7014b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:46:19.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d75e12640 1 --2- 172.21.15.107:0/156029863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d700c46a0 0x7f4d700c6fc0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c00c4d0 tx=0x7f4d6c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:19.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d667fc640 1 -- 172.21.15.107:0/156029863 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c005cf0 con 0x7f4d700c46a0 2023-12-04T05:46:19.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d667fc640 1 -- 172.21.15.107:0/156029863 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d6c005e50 con 0x7f4d700c46a0 2023-12-04T05:46:19.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d78076640 1 -- 172.21.15.107:0/156029863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d700ca1c0 con 0x7f4d700c46a0 2023-12-04T05:46:19.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d667fc640 1 -- 172.21.15.107:0/156029863 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c007290 con 0x7f4d700c46a0 2023-12-04T05:46:19.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.446+0000 7f4d78076640 1 -- 172.21.15.107:0/156029863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d700c94a0 con 0x7f4d700c46a0 2023-12-04T05:46:19.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.450+0000 7f4d78076640 1 -- 172.21.15.107:0/156029863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d700ca020 con 0x7f4d700c46a0 2023-12-04T05:46:19.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.450+0000 7f4d667fc640 1 -- 172.21.15.107:0/156029863 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4d6c0073f0 con 0x7f4d700c46a0 2023-12-04T05:46:19.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.450+0000 7f4d667fc640 1 --2- 172.21.15.107:0/156029863 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4d540668d0 0x7f4d54068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:19.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.450+0000 7f4d667fc640 1 -- 172.21.15.107:0/156029863 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f4d6c083ae0 con 0x7f4d700c46a0 2023-12-04T05:46:19.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.450+0000 7f4d75611640 1 --2- 172.21.15.107:0/156029863 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4d540668d0 0x7f4d54068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:19.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.450+0000 7f4d75611640 1 --2- 172.21.15.107:0/156029863 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4d540668d0 0x7f4d54068d90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4d600b0000 tx=0x7f4d600a8070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:19.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.454+0000 7f4d667fc640 1 -- 172.21.15.107:0/156029863 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4d700ca020 con 0x7f4d700c46a0 2023-12-04T05:46:19.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.642+0000 7f4d78076640 1 -- 172.21.15.107:0/156029863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d701f34c0 con 0x7f4d700c46a0 2023-12-04T05:46:19.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.642+0000 7f4d667fc640 1 -- 172.21.15.107:0/156029863 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f4d701f34c0 con 0x7f4d700c46a0 2023-12-04T05:46:19.644 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:19.645 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:19.645 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:46:19.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.642+0000 7f4d78076640 1 -- 172.21.15.107:0/156029863 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4d540668d0 msgr2=0x7f4d54068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:19.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.642+0000 7f4d78076640 1 --2- 172.21.15.107:0/156029863 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4d540668d0 0x7f4d54068d90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4d600b0000 tx=0x7f4d600a8070 comp rx=0 tx=0).stop 2023-12-04T05:46:19.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.642+0000 7f4d78076640 1 -- 172.21.15.107:0/156029863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d700c46a0 msgr2=0x7f4d700c6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:19.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.642+0000 7f4d78076640 1 --2- 172.21.15.107:0/156029863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d700c46a0 0x7f4d700c6fc0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c00c4d0 tx=0x7f4d6c00c990 comp rx=0 tx=0).stop 2023-12-04T05:46:19.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.642+0000 7f4d78076640 1 -- 172.21.15.107:0/156029863 shutdown_connections 2023-12-04T05:46:19.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.642+0000 7f4d78076640 1 --2- 172.21.15.107:0/156029863 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4d540668d0 0x7f4d54068d90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:19.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.642+0000 7f4d78076640 1 --2- 172.21.15.107:0/156029863 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d70157180 0x7f4d7014b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:19.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.642+0000 7f4d78076640 1 --2- 172.21.15.107:0/156029863 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4d700c4a80 0x7f4d7014b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:19.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.642+0000 7f4d78076640 1 --2- 172.21.15.107:0/156029863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d700c46a0 0x7f4d700c6fc0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:19.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.642+0000 7f4d78076640 1 -- 172.21.15.107:0/156029863 >> 172.21.15.107:0/156029863 conn(0x7f4d700c7ff0 msgr2=0x7f4d7014dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:19.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.642+0000 7f4d78076640 1 -- 172.21.15.107:0/156029863 shutdown_connections 2023-12-04T05:46:19.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:19.642+0000 7f4d78076640 1 -- 172.21.15.107:0/156029863 wait complete. 2023-12-04T05:46:19.664 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:46:21.417 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~40s 2023-12-04T05:46:21.417 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~154s 2023-12-04T05:46:21.417 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~195s 2023-12-04T05:46:21.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:21.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.794+0000 7f9dd8b55640 1 Processor -- start 2023-12-04T05:46:21.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.794+0000 7f9dd8b55640 1 -- start start 2023-12-04T05:46:21.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.794+0000 7f9dd8b55640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9dd4154020 0x7f9dd415a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:21.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.794+0000 7f9dd8b55640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9dd40d01c0 0x7f9dd4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:21.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.794+0000 7f9dd8b55640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9dd4072220 0x7f9dd40c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:21.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.794+0000 7f9dd8b55640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd40ca230 con 0x7f9dd4154020 2023-12-04T05:46:21.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.794+0000 7f9dd8b55640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd40c6770 con 0x7f9dd40d01c0 2023-12-04T05:46:21.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.794+0000 7f9dd8b55640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd415ad50 con 0x7f9dd4072220 2023-12-04T05:46:21.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.794+0000 7f9dd259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9dd4154020 0x7f9dd415a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:21.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.794+0000 7f9dd259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9dd4154020 0x7f9dd415a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53316/0 (socket says 172.21.15.107:53316) 2023-12-04T05:46:21.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.794+0000 7f9dd259c640 1 -- 172.21.15.107:0/3259815303 learned_addr learned my addr 172.21.15.107:0/3259815303 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:21.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.794+0000 7f9dd2d9d640 1 --2- 172.21.15.107:0/3259815303 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9dd4072220 0x7f9dd40c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:21.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.794+0000 7f9dcbfff640 1 --2- 172.21.15.107:0/3259815303 >> v2:172.21.15.195:3300/0 conn(0x7f9dd40d01c0 0x7f9dd4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:21.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.794+0000 7f9dd259c640 1 -- 172.21.15.107:0/3259815303 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9dd4072220 msgr2=0x7f9dd40c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:21.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.794+0000 7f9dd259c640 1 --2- 172.21.15.107:0/3259815303 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9dd4072220 0x7f9dd40c4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:21.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.794+0000 7f9dd259c640 1 -- 172.21.15.107:0/3259815303 >> v2:172.21.15.195:3300/0 conn(0x7f9dd40d01c0 msgr2=0x7f9dd4071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:21.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.794+0000 7f9dd259c640 1 --2- 172.21.15.107:0/3259815303 >> v2:172.21.15.195:3300/0 conn(0x7f9dd40d01c0 0x7f9dd4071ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:21.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.794+0000 7f9dd259c640 1 -- 172.21.15.107:0/3259815303 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd415d710 con 0x7f9dd4154020 2023-12-04T05:46:21.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.794+0000 7f9dcbfff640 1 --2- 172.21.15.107:0/3259815303 >> v2:172.21.15.195:3300/0 conn(0x7f9dd40d01c0 0x7f9dd4071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:46:21.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd259c640 1 --2- 172.21.15.107:0/3259815303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dd4154020 0x7f9dd415a810 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f9dd406b6d0 tx=0x7f9dc401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d214b22f512d3836 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:21.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd1d9b640 1 -- 172.21.15.107:0/3259815303 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc40a89b0 con 0x7f9dd4154020 2023-12-04T05:46:21.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd1d9b640 1 -- 172.21.15.107:0/3259815303 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc40a8b10 con 0x7f9dd4154020 2023-12-04T05:46:21.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd1d9b640 1 -- 172.21.15.107:0/3259815303 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc40a8ea0 con 0x7f9dd4154020 2023-12-04T05:46:21.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd8b55640 1 -- 172.21.15.107:0/3259815303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dd4154020 msgr2=0x7f9dd415a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:21.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd8b55640 1 --2- 172.21.15.107:0/3259815303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dd4154020 0x7f9dd415a810 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f9dd406b6d0 tx=0x7f9dc401ecf0 comp rx=0 tx=0).stop 2023-12-04T05:46:21.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd8b55640 1 -- 172.21.15.107:0/3259815303 shutdown_connections 2023-12-04T05:46:21.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd8b55640 1 --2- 172.21.15.107:0/3259815303 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9dd4072220 0x7f9dd40c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:21.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd8b55640 1 --2- 172.21.15.107:0/3259815303 >> v2:172.21.15.195:3300/0 conn(0x7f9dd40d01c0 0x7f9dd4071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:21.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd8b55640 1 --2- 172.21.15.107:0/3259815303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dd4154020 0x7f9dd415a810 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:21.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd8b55640 1 -- 172.21.15.107:0/3259815303 >> 172.21.15.107:0/3259815303 conn(0x7f9dd406c6c0 msgr2=0x7f9dd40cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:21.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd8b55640 1 -- 172.21.15.107:0/3259815303 shutdown_connections 2023-12-04T05:46:21.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd8b55640 1 -- 172.21.15.107:0/3259815303 wait complete. 2023-12-04T05:46:21.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd8b55640 1 Processor -- start 2023-12-04T05:46:21.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd8b55640 1 -- start start 2023-12-04T05:46:21.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd8b55640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9dd4072220 0x7f9dd41ef980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:21.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd8b55640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9dd40d01c0 0x7f9dd414d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:21.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd8b55640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dd4154020 0x7f9dd415b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:21.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd8b55640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd415d280 con 0x7f9dd4154020 2023-12-04T05:46:21.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd8b55640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd41545a0 con 0x7f9dd4072220 2023-12-04T05:46:21.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd8b55640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd4154880 con 0x7f9dd40d01c0 2023-12-04T05:46:21.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dd4154020 0x7f9dd415b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:21.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dd4154020 0x7f9dd415b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53322/0 (socket says 172.21.15.107:53322) 2023-12-04T05:46:21.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.798+0000 7f9dd2d9d640 1 -- 172.21.15.107:0/3787219792 learned_addr learned my addr 172.21.15.107:0/3787219792 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:21.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.802+0000 7f9dd259c640 1 --2- 172.21.15.107:0/3787219792 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9dd4072220 0x7f9dd41ef980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:21.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.802+0000 7f9dcbfff640 1 --2- 172.21.15.107:0/3787219792 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9dd40d01c0 0x7f9dd414d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:21.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.802+0000 7f9dd2d9d640 1 -- 172.21.15.107:0/3787219792 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9dd40d01c0 msgr2=0x7f9dd414d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:21.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.802+0000 7f9dd2d9d640 1 --2- 172.21.15.107:0/3787219792 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9dd40d01c0 0x7f9dd414d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:21.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.802+0000 7f9dd2d9d640 1 -- 172.21.15.107:0/3787219792 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9dd4072220 msgr2=0x7f9dd41ef980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:21.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.802+0000 7f9dd2d9d640 1 --2- 172.21.15.107:0/3787219792 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9dd4072220 0x7f9dd41ef980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:21.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.802+0000 7f9dd2d9d640 1 -- 172.21.15.107:0/3787219792 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc40094d0 con 0x7f9dd4154020 2023-12-04T05:46:21.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.802+0000 7f9dd2d9d640 1 --2- 172.21.15.107:0/3787219792 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dd4154020 0x7f9dd415b670 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc007e60 tx=0x7f9dcc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:21.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.802+0000 7f9dcaffd640 1 -- 172.21.15.107:0/3787219792 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc014620 con 0x7f9dd4154020 2023-12-04T05:46:21.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.802+0000 7f9dcaffd640 1 -- 172.21.15.107:0/3787219792 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dcc00ee90 con 0x7f9dd4154020 2023-12-04T05:46:21.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.802+0000 7f9dcaffd640 1 -- 172.21.15.107:0/3787219792 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc0149e0 con 0x7f9dd4154020 2023-12-04T05:46:21.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.802+0000 7f9dd8b55640 1 -- 172.21.15.107:0/3787219792 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd4154b60 con 0x7f9dd4154020 2023-12-04T05:46:21.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.802+0000 7f9dd8b55640 1 -- 172.21.15.107:0/3787219792 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd4155120 con 0x7f9dd4154020 2023-12-04T05:46:21.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.802+0000 7f9dd8b55640 1 -- 172.21.15.107:0/3787219792 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd4067870 con 0x7f9dd4154020 2023-12-04T05:46:21.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.802+0000 7f9dcaffd640 1 -- 172.21.15.107:0/3787219792 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9dcc014b40 con 0x7f9dd4154020 2023-12-04T05:46:21.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.802+0000 7f9dcaffd640 1 --2- 172.21.15.107:0/3787219792 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9db80668d0 0x7f9db8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:21.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.802+0000 7f9dcaffd640 1 -- 172.21.15.107:0/3787219792 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f9dcc084fe0 con 0x7f9dd4154020 2023-12-04T05:46:21.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.802+0000 7f9dd259c640 1 --2- 172.21.15.107:0/3787219792 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9db80668d0 0x7f9db8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:21.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.802+0000 7f9dd259c640 1 --2- 172.21.15.107:0/3787219792 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9db80668d0 0x7f9db8068d90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9dc40b2000 tx=0x7f9dc4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:21.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.810+0000 7f9dcaffd640 1 -- 172.21.15.107:0/3787219792 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9dcc051fc0 con 0x7f9dd4154020 2023-12-04T05:46:21.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.994+0000 7f9dd8b55640 1 -- 172.21.15.107:0/3787219792 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9dd415b3d0 con 0x7f9dd4154020 2023-12-04T05:46:21.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.994+0000 7f9dcaffd640 1 -- 172.21.15.107:0/3787219792 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f9dcc0555e0 con 0x7f9dd4154020 2023-12-04T05:46:21.998 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:21.999 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:21.999 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:46:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.998+0000 7f9dd8b55640 1 -- 172.21.15.107:0/3787219792 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9db80668d0 msgr2=0x7f9db8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.998+0000 7f9dd8b55640 1 --2- 172.21.15.107:0/3787219792 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9db80668d0 0x7f9db8068d90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9dc40b2000 tx=0x7f9dc4008040 comp rx=0 tx=0).stop 2023-12-04T05:46:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.998+0000 7f9dd8b55640 1 -- 172.21.15.107:0/3787219792 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dd4154020 msgr2=0x7f9dd415b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.998+0000 7f9dd8b55640 1 --2- 172.21.15.107:0/3787219792 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dd4154020 0x7f9dd415b670 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc007e60 tx=0x7f9dcc00e4b0 comp rx=0 tx=0).stop 2023-12-04T05:46:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.998+0000 7f9dd8b55640 1 -- 172.21.15.107:0/3787219792 shutdown_connections 2023-12-04T05:46:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.998+0000 7f9dd8b55640 1 --2- 172.21.15.107:0/3787219792 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9db80668d0 0x7f9db8068d90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.998+0000 7f9dd8b55640 1 --2- 172.21.15.107:0/3787219792 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dd4154020 0x7f9dd415b670 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.998+0000 7f9dd8b55640 1 --2- 172.21.15.107:0/3787219792 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9dd40d01c0 0x7f9dd414d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.998+0000 7f9dd8b55640 1 --2- 172.21.15.107:0/3787219792 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9dd4072220 0x7f9dd41ef980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.998+0000 7f9dd8b55640 1 -- 172.21.15.107:0/3787219792 >> 172.21.15.107:0/3787219792 conn(0x7f9dd406c6c0 msgr2=0x7f9dd414f990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.998+0000 7f9dd8b55640 1 -- 172.21.15.107:0/3787219792 shutdown_connections 2023-12-04T05:46:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:21.998+0000 7f9dd8b55640 1 -- 172.21.15.107:0/3787219792 wait complete. 2023-12-04T05:46:22.017 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:46:24.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:24.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.146+0000 7f2dfdb20640 1 Processor -- start 2023-12-04T05:46:24.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.146+0000 7f2dfdb20640 1 -- start start 2023-12-04T05:46:24.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.146+0000 7f2dfdb20640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2df8154020 0x7f2df815a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:24.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.146+0000 7f2dfdb20640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2df80d01c0 0x7f2df8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:24.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.146+0000 7f2dfdb20640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df815ad50 con 0x7f2df8072220 2023-12-04T05:46:24.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.146+0000 7f2dfdb20640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2df80ca230 con 0x7f2df80d01c0 2023-12-04T05:46:24.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.146+0000 7f2dfdb20640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df80c6770 con 0x7f2df8154020 2023-12-04T05:46:24.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.146+0000 7f2df6ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2df80d01c0 0x7f2df8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:24.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.146+0000 7f2df7fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2df8072220 0x7f2df80c4c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37978/0 (socket says 172.21.15.107:37978) 2023-12-04T05:46:24.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.146+0000 7f2df7fff640 1 -- 172.21.15.107:0/4253987437 learned_addr learned my addr 172.21.15.107:0/4253987437 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:46:24.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.146+0000 7f2df77fe640 1 --2- 172.21.15.107:0/4253987437 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2df8154020 0x7f2df815a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:24.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.146+0000 7f2df67fc640 1 -- 172.21.15.107:0/4253987437 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857496673 0 0) 0x7f2df815ad50 con 0x7f2df8072220 2023-12-04T05:46:24.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.146+0000 7f2df67fc640 1 -- 172.21.15.107:0/4253987437 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2de0003530 con 0x7f2df8072220 2023-12-04T05:46:24.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.146+0000 7f2df6ffd640 1 -- 172.21.15.107:0/4253987437 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2df8154020 msgr2=0x7f2df815a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:24.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.146+0000 7f2df6ffd640 1 --2- 172.21.15.107:0/4253987437 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2df8154020 0x7f2df815a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:24.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.146+0000 7f2df6ffd640 1 -- 172.21.15.107:0/4253987437 >> v1:172.21.15.107:6789/0 conn(0x7f2df8072220 legacy=0x7f2df80c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:24.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.146+0000 7f2df6ffd640 1 -- 172.21.15.107:0/4253987437 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df815d710 con 0x7f2df80d01c0 2023-12-04T05:46:24.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.150+0000 7f2df6ffd640 1 --2- 172.21.15.107:0/4253987437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2df80d01c0 0x7f2df8071ce0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2dec0097c0 tx=0x7f2dec01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bc6c33f5d5e9f0f server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:24.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.150+0000 7f2df67fc640 1 -- 172.21.15.107:0/4253987437 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec0a8970 con 0x7f2df80d01c0 2023-12-04T05:46:24.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.150+0000 7f2df67fc640 1 -- 172.21.15.107:0/4253987437 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dec0a8ad0 con 0x7f2df80d01c0 2023-12-04T05:46:24.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.150+0000 7f2df67fc640 1 -- 172.21.15.107:0/4253987437 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec0a8e60 con 0x7f2df80d01c0 2023-12-04T05:46:24.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.150+0000 7f2dfdb20640 1 -- 172.21.15.107:0/4253987437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2df80d01c0 msgr2=0x7f2df8071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:24.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.150+0000 7f2dfdb20640 1 --2- 172.21.15.107:0/4253987437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2df80d01c0 0x7f2df8071ce0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2dec0097c0 tx=0x7f2dec01ec70 comp rx=0 tx=0).stop 2023-12-04T05:46:24.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.150+0000 7f2dfdb20640 1 -- 172.21.15.107:0/4253987437 shutdown_connections 2023-12-04T05:46:24.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.150+0000 7f2dfdb20640 1 --2- 172.21.15.107:0/4253987437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2df80d01c0 0x7f2df8071ce0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:24.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.150+0000 7f2dfdb20640 1 --2- 172.21.15.107:0/4253987437 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2df8154020 0x7f2df815a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:24.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.150+0000 7f2dfdb20640 1 -- 172.21.15.107:0/4253987437 >> 172.21.15.107:0/4253987437 conn(0x7f2df806c6c0 msgr2=0x7f2df80cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:24.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.150+0000 7f2dfdb20640 1 -- 172.21.15.107:0/4253987437 shutdown_connections 2023-12-04T05:46:24.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.150+0000 7f2dfdb20640 1 -- 172.21.15.107:0/4253987437 wait complete. 2023-12-04T05:46:24.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.150+0000 7f2dfdb20640 1 Processor -- start 2023-12-04T05:46:24.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.150+0000 7f2dfdb20640 1 -- start start 2023-12-04T05:46:24.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2dfdb20640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2df8072220 0x7f2df81f2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:24.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2dfdb20640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2df80d01c0 0x7f2df80c8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:24.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2dfdb20640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2df8154020 0x7f2df80ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:24.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2dfdb20640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df80c8d50 con 0x7f2df8154020 2023-12-04T05:46:24.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2dfdb20640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df814fdd0 con 0x7f2df8072220 2023-12-04T05:46:24.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2dfdb20640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df814e2d0 con 0x7f2df80d01c0 2023-12-04T05:46:24.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2df7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2df8154020 0x7f2df80ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:24.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2df7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2df8154020 0x7f2df80ceef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53338/0 (socket says 172.21.15.107:53338) 2023-12-04T05:46:24.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2df7fff640 1 -- 172.21.15.107:0/1318402029 learned_addr learned my addr 172.21.15.107:0/1318402029 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:24.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2df6ffd640 1 --2- 172.21.15.107:0/1318402029 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2df80d01c0 0x7f2df80c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:24.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2df7fff640 1 -- 172.21.15.107:0/1318402029 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2df80d01c0 msgr2=0x7f2df80c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:24.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2df7fff640 1 --2- 172.21.15.107:0/1318402029 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2df80d01c0 0x7f2df80c8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:24.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2df7fff640 1 -- 172.21.15.107:0/1318402029 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2df8072220 msgr2=0x7f2df81f2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:24.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2df7fff640 1 --2- 172.21.15.107:0/1318402029 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2df8072220 0x7f2df81f2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:24.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2df7fff640 1 -- 172.21.15.107:0/1318402029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dec009420 con 0x7f2df8154020 2023-12-04T05:46:24.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2df7fff640 1 --2- 172.21.15.107:0/1318402029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2df8154020 0x7f2df80ceef0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2df000cb70 tx=0x7f2df000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:24.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2ddffff640 1 -- 172.21.15.107:0/1318402029 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df000ace0 con 0x7f2df8154020 2023-12-04T05:46:24.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2ddffff640 1 -- 172.21.15.107:0/1318402029 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df000ae40 con 0x7f2df8154020 2023-12-04T05:46:24.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2dfdb20640 1 -- 172.21.15.107:0/1318402029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df80c9540 con 0x7f2df8154020 2023-12-04T05:46:24.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2ddffff640 1 -- 172.21.15.107:0/1318402029 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0012620 con 0x7f2df8154020 2023-12-04T05:46:24.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2dfdb20640 1 -- 172.21.15.107:0/1318402029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df814e520 con 0x7f2df8154020 2023-12-04T05:46:24.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2dfdb20640 1 -- 172.21.15.107:0/1318402029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2df80c8530 con 0x7f2df8154020 2023-12-04T05:46:24.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2ddffff640 1 -- 172.21.15.107:0/1318402029 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f2df0007070 con 0x7f2df8154020 2023-12-04T05:46:24.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2ddffff640 1 --2- 172.21.15.107:0/1318402029 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2de40668d0 0x7f2de4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:24.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2df77fe640 1 --2- 172.21.15.107:0/1318402029 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2de40668d0 0x7f2de4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:24.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2ddffff640 1 -- 172.21.15.107:0/1318402029 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f2df00846a0 con 0x7f2df8154020 2023-12-04T05:46:24.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.154+0000 7f2df77fe640 1 --2- 172.21.15.107:0/1318402029 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2de40668d0 0x7f2de4068d90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2de800b850 tx=0x7f2de8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:24.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.158+0000 7f2ddffff640 1 -- 172.21.15.107:0/1318402029 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2df80c8530 con 0x7f2df8154020 2023-12-04T05:46:24.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.346+0000 7f2dfdb20640 1 -- 172.21.15.107:0/1318402029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2df80c8530 con 0x7f2df8154020 2023-12-04T05:46:24.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.346+0000 7f2ddffff640 1 -- 172.21.15.107:0/1318402029 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f2df80c8530 con 0x7f2df8154020 2023-12-04T05:46:24.348 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:24.348 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:24.348 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:46:24.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.346+0000 7f2dfdb20640 1 -- 172.21.15.107:0/1318402029 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2de40668d0 msgr2=0x7f2de4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:24.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.346+0000 7f2dfdb20640 1 --2- 172.21.15.107:0/1318402029 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2de40668d0 0x7f2de4068d90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2de800b850 tx=0x7f2de8008040 comp rx=0 tx=0).stop 2023-12-04T05:46:24.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.346+0000 7f2dfdb20640 1 -- 172.21.15.107:0/1318402029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2df8154020 msgr2=0x7f2df80ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:24.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.346+0000 7f2dfdb20640 1 --2- 172.21.15.107:0/1318402029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2df8154020 0x7f2df80ceef0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2df000cb70 tx=0x7f2df000a430 comp rx=0 tx=0).stop 2023-12-04T05:46:24.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.346+0000 7f2dfdb20640 1 -- 172.21.15.107:0/1318402029 shutdown_connections 2023-12-04T05:46:24.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.346+0000 7f2dfdb20640 1 --2- 172.21.15.107:0/1318402029 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f2de40668d0 0x7f2de4068d90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:24.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.346+0000 7f2dfdb20640 1 --2- 172.21.15.107:0/1318402029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2df8154020 0x7f2df80ceef0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:24.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.346+0000 7f2dfdb20640 1 --2- 172.21.15.107:0/1318402029 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f2df80d01c0 0x7f2df80c8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:24.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.346+0000 7f2dfdb20640 1 --2- 172.21.15.107:0/1318402029 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2df8072220 0x7f2df81f2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:24.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.346+0000 7f2dfdb20640 1 -- 172.21.15.107:0/1318402029 >> 172.21.15.107:0/1318402029 conn(0x7f2df806c6c0 msgr2=0x7f2df80cb1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:24.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.346+0000 7f2dfdb20640 1 -- 172.21.15.107:0/1318402029 shutdown_connections 2023-12-04T05:46:24.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:24.346+0000 7f2dfdb20640 1 -- 172.21.15.107:0/1318402029 wait complete. 2023-12-04T05:46:24.367 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:46:26.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:26.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54b4ae0640 1 Processor -- start 2023-12-04T05:46:26.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54b4ae0640 1 -- start start 2023-12-04T05:46:26.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54b4ae0640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54b00724f0 0x7f54b0158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:26.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54b4ae0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b0158da0 con 0x7f54b00c56f0 2023-12-04T05:46:26.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54b4ae0640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b01583d0 con 0x7f54b0072110 2023-12-04T05:46:26.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54b4ae0640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b00c6170 con 0x7f54b00724f0 2023-12-04T05:46:26.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54aed9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f54b00c56f0 0x7f54b0140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37984/0 (socket says 172.21.15.107:37984) 2023-12-04T05:46:26.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54aed9d640 1 -- 172.21.15.107:0/1336478019 learned_addr learned my addr 172.21.15.107:0/1336478019 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:46:26.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54add9b640 1 --2- 172.21.15.107:0/1336478019 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54b00724f0 0x7f54b0158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:26.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54ad59a640 1 -- 172.21.15.107:0/1336478019 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313108121 0 0) 0x7f54a80014a0 con 0x7f54b00c56f0 2023-12-04T05:46:26.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54ad59a640 1 -- 172.21.15.107:0/1336478019 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5498003530 con 0x7f54b00c56f0 2023-12-04T05:46:26.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54ad59a640 1 -- 172.21.15.107:0/1336478019 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2572056196 0 0) 0x7f5498003530 con 0x7f54b00c56f0 2023-12-04T05:46:26.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54ad59a640 1 -- 172.21.15.107:0/1336478019 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54b00724f0 msgr2=0x7f54b0158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:26.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54ad59a640 1 --2- 172.21.15.107:0/1336478019 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54b00724f0 0x7f54b0158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:26.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54ad59a640 1 -- 172.21.15.107:0/1336478019 >> v1:172.21.15.195:6789/0 conn(0x7f54b0072110 legacy=0x7f54b00c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:26.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54ad59a640 1 -- 172.21.15.107:0/1336478019 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b0157da0 con 0x7f54b00c56f0 2023-12-04T05:46:26.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54ad59a640 1 -- 172.21.15.107:0/1336478019 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f54a8001b90 con 0x7f54b00c56f0 2023-12-04T05:46:26.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54ad59a640 1 -- 172.21.15.107:0/1336478019 >> v1:172.21.15.107:6789/0 conn(0x7f54b00c56f0 legacy=0x7f54b0140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:26.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54ad59a640 1 --2- 172.21.15.107:0/1336478019 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54980039b0 0x7f5498003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:26.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54ad59a640 1 --2- 172.21.15.107:0/1336478019 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54b00724f0 0x7f54980047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:26.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54ad59a640 1 --2- 172.21.15.107:0/1336478019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5498004d10 0x7f5498005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:26.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54ad59a640 1 -- 172.21.15.107:0/1336478019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b00c6170 con 0x7f5498004d10 2023-12-04T05:46:26.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54ad59a640 1 -- 172.21.15.107:0/1336478019 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549800bb30 con 0x7f54980039b0 2023-12-04T05:46:26.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54ad59a640 1 -- 172.21.15.107:0/1336478019 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f549800bcf0 con 0x7f54b00724f0 2023-12-04T05:46:26.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54aed9d640 1 --2- 172.21.15.107:0/1336478019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5498004d10 0x7f5498005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:26.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54add9b640 1 --2- 172.21.15.107:0/1336478019 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54b00724f0 0x7f54980047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:26.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54ae59c640 1 --2- 172.21.15.107:0/1336478019 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54980039b0 0x7f5498003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:26.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54aed9d640 1 -- 172.21.15.107:0/1336478019 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54b00724f0 msgr2=0x7f54980047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:26.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54aed9d640 1 --2- 172.21.15.107:0/1336478019 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54b00724f0 0x7f54980047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:26.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54aed9d640 1 -- 172.21.15.107:0/1336478019 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54980039b0 msgr2=0x7f5498003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:26.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54aed9d640 1 --2- 172.21.15.107:0/1336478019 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54980039b0 0x7f5498003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:26.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54aed9d640 1 -- 172.21.15.107:0/1336478019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5498003530 con 0x7f5498004d10 2023-12-04T05:46:26.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.502+0000 7f54ae59c640 1 --2- 172.21.15.107:0/1336478019 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54980039b0 0x7f5498003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:46:26.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.506+0000 7f54aed9d640 1 --2- 172.21.15.107:0/1336478019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5498004d10 0x7f5498005190 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f54a80013e0 tx=0x7f54a801d200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99fb86183857c35 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:26.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.506+0000 7f54ad59a640 1 -- 172.21.15.107:0/1336478019 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a80ad630 con 0x7f5498004d10 2023-12-04T05:46:26.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.506+0000 7f54ad59a640 1 -- 172.21.15.107:0/1336478019 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a80a8070 con 0x7f5498004d10 2023-12-04T05:46:26.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.506+0000 7f54ad59a640 1 -- 172.21.15.107:0/1336478019 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a80ad9f0 con 0x7f5498004d10 2023-12-04T05:46:26.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.506+0000 7f54b4ae0640 1 -- 172.21.15.107:0/1336478019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5498004d10 msgr2=0x7f5498005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:26.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.506+0000 7f54b4ae0640 1 --2- 172.21.15.107:0/1336478019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5498004d10 0x7f5498005190 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f54a80013e0 tx=0x7f54a801d200 comp rx=0 tx=0).stop 2023-12-04T05:46:26.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.506+0000 7f54aed9d640 1 -- 172.21.15.107:0/1336478019 reap_dead start 2023-12-04T05:46:26.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.506+0000 7f54b4ae0640 1 -- 172.21.15.107:0/1336478019 shutdown_connections 2023-12-04T05:46:26.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.506+0000 7f54b4ae0640 1 -- 172.21.15.107:0/1336478019 >> 172.21.15.107:0/1336478019 conn(0x7f54b00c7ff0 msgr2=0x7f54b00c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:26.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.506+0000 7f54b4ae0640 1 -- 172.21.15.107:0/1336478019 shutdown_connections 2023-12-04T05:46:26.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.506+0000 7f54b4ae0640 1 -- 172.21.15.107:0/1336478019 wait complete. 2023-12-04T05:46:26.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.506+0000 7f54b4ae0640 1 Processor -- start 2023-12-04T05:46:26.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.506+0000 7f54b4ae0640 1 -- start start 2023-12-04T05:46:26.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.506+0000 7f54b4ae0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54b014a4c0 0x7f54b00c9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:26.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.506+0000 7f54b4ae0640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54b0158f60 0x7f54b00ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:26.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.506+0000 7f54b4ae0640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54b0154a00 0x7f54b0154e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:26.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.506+0000 7f54b4ae0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b0153560 con 0x7f54b014a4c0 2023-12-04T05:46:26.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.506+0000 7f54b4ae0640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b01553c0 con 0x7f54b0158f60 2023-12-04T05:46:26.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.506+0000 7f54b4ae0640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b0152690 con 0x7f54b0154a00 2023-12-04T05:46:26.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.506+0000 7f54ae59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54b014a4c0 0x7f54b00c9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:26.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.506+0000 7f54ae59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54b014a4c0 0x7f54b00c9f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53360/0 (socket says 172.21.15.107:53360) 2023-12-04T05:46:26.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.506+0000 7f54ae59c640 1 -- 172.21.15.107:0/1821313485 learned_addr learned my addr 172.21.15.107:0/1821313485 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:26.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.510+0000 7f54aed9d640 1 --2- 172.21.15.107:0/1821313485 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54b0154a00 0x7f54b0154e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:26.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.510+0000 7f54add9b640 1 --2- 172.21.15.107:0/1821313485 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54b0158f60 0x7f54b00ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:26.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.510+0000 7f54ae59c640 1 -- 172.21.15.107:0/1821313485 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54b0154a00 msgr2=0x7f54b0154e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:26.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.510+0000 7f54ae59c640 1 --2- 172.21.15.107:0/1821313485 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54b0154a00 0x7f54b0154e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:26.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.510+0000 7f54ae59c640 1 -- 172.21.15.107:0/1821313485 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54b0158f60 msgr2=0x7f54b00ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:26.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.510+0000 7f54ae59c640 1 --2- 172.21.15.107:0/1821313485 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54b0158f60 0x7f54b00ca450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:26.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.510+0000 7f54ae59c640 1 -- 172.21.15.107:0/1821313485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b0153210 con 0x7f54b014a4c0 2023-12-04T05:46:26.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.510+0000 7f54ae59c640 1 --2- 172.21.15.107:0/1821313485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54b014a4c0 0x7f54b00c9f10 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f54a000d540 tx=0x7f54a000da00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:26.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.510+0000 7f549effd640 1 -- 172.21.15.107:0/1821313485 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a0005480 con 0x7f54b014a4c0 2023-12-04T05:46:26.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.510+0000 7f549effd640 1 -- 172.21.15.107:0/1821313485 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a0007030 con 0x7f54b014a4c0 2023-12-04T05:46:26.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.510+0000 7f54b4ae0640 1 -- 172.21.15.107:0/1821313485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b01555b0 con 0x7f54b014a4c0 2023-12-04T05:46:26.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.510+0000 7f549effd640 1 -- 172.21.15.107:0/1821313485 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a0007420 con 0x7f54b014a4c0 2023-12-04T05:46:26.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.510+0000 7f54b4ae0640 1 -- 172.21.15.107:0/1821313485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b0152bc0 con 0x7f54b014a4c0 2023-12-04T05:46:26.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.510+0000 7f54b4ae0640 1 -- 172.21.15.107:0/1821313485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54b0067870 con 0x7f54b014a4c0 2023-12-04T05:46:26.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.510+0000 7f549effd640 1 -- 172.21.15.107:0/1821313485 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f54a0007580 con 0x7f54b014a4c0 2023-12-04T05:46:26.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.510+0000 7f549effd640 1 --2- 172.21.15.107:0/1821313485 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f54900668d0 0x7f5490068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:26.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.510+0000 7f549effd640 1 -- 172.21.15.107:0/1821313485 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f54a00840a0 con 0x7f54b014a4c0 2023-12-04T05:46:26.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.510+0000 7f54add9b640 1 --2- 172.21.15.107:0/1821313485 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f54900668d0 0x7f5490068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:26.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.510+0000 7f54add9b640 1 --2- 172.21.15.107:0/1821313485 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f54900668d0 0x7f5490068d90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f54b0158d70 tx=0x7f54a4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:26.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.514+0000 7f549effd640 1 -- 172.21.15.107:0/1821313485 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f54a0050ff0 con 0x7f54b014a4c0 2023-12-04T05:46:26.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.694+0000 7f54b4ae0640 1 -- 172.21.15.107:0/1821313485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54b00cb240 con 0x7f54b014a4c0 2023-12-04T05:46:26.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.694+0000 7f549effd640 1 -- 172.21.15.107:0/1821313485 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f54a0019090 con 0x7f54b014a4c0 2023-12-04T05:46:26.697 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:26.697 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:26.697 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:46:26.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.694+0000 7f54b4ae0640 1 -- 172.21.15.107:0/1821313485 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f54900668d0 msgr2=0x7f5490068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:26.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.694+0000 7f54b4ae0640 1 --2- 172.21.15.107:0/1821313485 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f54900668d0 0x7f5490068d90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f54b0158d70 tx=0x7f54a4008040 comp rx=0 tx=0).stop 2023-12-04T05:46:26.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.694+0000 7f54b4ae0640 1 -- 172.21.15.107:0/1821313485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54b014a4c0 msgr2=0x7f54b00c9f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:26.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.694+0000 7f54b4ae0640 1 --2- 172.21.15.107:0/1821313485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54b014a4c0 0x7f54b00c9f10 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f54a000d540 tx=0x7f54a000da00 comp rx=0 tx=0).stop 2023-12-04T05:46:26.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.698+0000 7f54b4ae0640 1 -- 172.21.15.107:0/1821313485 shutdown_connections 2023-12-04T05:46:26.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.698+0000 7f54b4ae0640 1 --2- 172.21.15.107:0/1821313485 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f54900668d0 0x7f5490068d90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:26.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.698+0000 7f54b4ae0640 1 --2- 172.21.15.107:0/1821313485 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f54b0154a00 0x7f54b0154e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:26.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.698+0000 7f54b4ae0640 1 --2- 172.21.15.107:0/1821313485 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f54b0158f60 0x7f54b00ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:26.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.698+0000 7f54b4ae0640 1 --2- 172.21.15.107:0/1821313485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54b014a4c0 0x7f54b00c9f10 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:26.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.698+0000 7f54b4ae0640 1 -- 172.21.15.107:0/1821313485 >> 172.21.15.107:0/1821313485 conn(0x7f54b00c7ff0 msgr2=0x7f54b0155c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:26.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.698+0000 7f54b4ae0640 1 -- 172.21.15.107:0/1821313485 shutdown_connections 2023-12-04T05:46:26.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:26.698+0000 7f54b4ae0640 1 -- 172.21.15.107:0/1821313485 wait complete. 2023-12-04T05:46:26.715 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:46:28.033 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~46s 2023-12-04T05:46:28.034 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~160s 2023-12-04T05:46:28.034 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~201s 2023-12-04T05:46:28.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:28.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.842+0000 7fa0d6a31640 1 Processor -- start 2023-12-04T05:46:28.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.846+0000 7fa0d6a31640 1 -- start start 2023-12-04T05:46:28.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.846+0000 7fa0d6a31640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa0d0151180 0x7fa0d00c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:28.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.846+0000 7fa0d6a31640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa0d00714f0 0x7fa0d0071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:28.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.846+0000 7fa0d6a31640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d00700c0 con 0x7fa0d0070380 2023-12-04T05:46:28.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.846+0000 7fa0d6a31640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0d0071e60 con 0x7fa0d0151180 2023-12-04T05:46:28.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.846+0000 7fa0d6a31640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0d0156df0 con 0x7fa0d00714f0 2023-12-04T05:46:28.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.846+0000 7fa0d4fce640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa0d0070380 0x7fa0d0070780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33508/0 (socket says 172.21.15.107:33508) 2023-12-04T05:46:28.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.846+0000 7fa0cffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa0d0151180 0x7fa0d00c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:28.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.846+0000 7fa0d4fce640 1 -- 172.21.15.107:0/1195718187 learned_addr learned my addr 172.21.15.107:0/1195718187 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:46:28.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.846+0000 7fa0c7fff640 1 --2- 172.21.15.107:0/1195718187 >> v2:172.21.15.195:3300/0 conn(0x7fa0d00714f0 0x7fa0d0071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:28.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.846+0000 7fa0cffff640 1 -- 172.21.15.107:0/1195718187 >> v2:172.21.15.195:3300/0 conn(0x7fa0d00714f0 msgr2=0x7fa0d0071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:28.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.846+0000 7fa0cffff640 1 --2- 172.21.15.107:0/1195718187 >> v2:172.21.15.195:3300/0 conn(0x7fa0d00714f0 0x7fa0d0071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:28.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.846+0000 7fa0cffff640 1 -- 172.21.15.107:0/1195718187 >> v1:172.21.15.107:6789/0 conn(0x7fa0d0070380 legacy=0x7fa0d0070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:28.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.846+0000 7fa0cf7fe640 1 -- 172.21.15.107:0/1195718187 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233660329 0 0) 0x7fa0d00700c0 con 0x7fa0d0070380 2023-12-04T05:46:28.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.846+0000 7fa0cffff640 1 -- 172.21.15.107:0/1195718187 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d01570d0 con 0x7fa0d0151180 2023-12-04T05:46:28.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0cffff640 1 --2- 172.21.15.107:0/1195718187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0d0151180 0x7fa0d00c79e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fa0d006e5e0 tx=0x7fa0c001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=226a180a3cf8b77d server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:28.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0cf7fe640 1 -- 172.21.15.107:0/1195718187 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c00a8950 con 0x7fa0d0151180 2023-12-04T05:46:28.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0cf7fe640 1 -- 172.21.15.107:0/1195718187 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c00a8ab0 con 0x7fa0d0151180 2023-12-04T05:46:28.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0cf7fe640 1 -- 172.21.15.107:0/1195718187 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c00a8e40 con 0x7fa0d0151180 2023-12-04T05:46:28.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0d6a31640 1 -- 172.21.15.107:0/1195718187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0d0151180 msgr2=0x7fa0d00c79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:28.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0d6a31640 1 --2- 172.21.15.107:0/1195718187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0d0151180 0x7fa0d00c79e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fa0d006e5e0 tx=0x7fa0c001ec50 comp rx=0 tx=0).stop 2023-12-04T05:46:28.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0d6a31640 1 -- 172.21.15.107:0/1195718187 shutdown_connections 2023-12-04T05:46:28.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0d6a31640 1 --2- 172.21.15.107:0/1195718187 >> v2:172.21.15.195:3300/0 conn(0x7fa0d00714f0 0x7fa0d0071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:28.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0d6a31640 1 --2- 172.21.15.107:0/1195718187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0d0151180 0x7fa0d00c79e0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:28.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0d6a31640 1 -- 172.21.15.107:0/1195718187 >> 172.21.15.107:0/1195718187 conn(0x7fa0d00c7ff0 msgr2=0x7fa0d00c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:28.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0d6a31640 1 -- 172.21.15.107:0/1195718187 shutdown_connections 2023-12-04T05:46:28.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0d6a31640 1 -- 172.21.15.107:0/1195718187 wait complete. 2023-12-04T05:46:28.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0d6a31640 1 Processor -- start 2023-12-04T05:46:28.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0d6a31640 1 -- start start 2023-12-04T05:46:28.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0d6a31640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0d0070380 0x7fa0d00ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:28.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0d6a31640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa0d00714f0 0x7fa0d01eda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:28.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0d6a31640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa0d0151180 0x7fa0d01edf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:28.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0d6a31640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d015a180 con 0x7fa0d0070380 2023-12-04T05:46:28.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0d6a31640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d01f1ae0 con 0x7fa0d0151180 2023-12-04T05:46:28.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0d6a31640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d01f1f00 con 0x7fa0d00714f0 2023-12-04T05:46:28.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0cffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0d0070380 0x7fa0d00ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:28.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0cffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0d0070380 0x7fa0d00ca6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55062/0 (socket says 172.21.15.107:55062) 2023-12-04T05:46:28.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.850+0000 7fa0cffff640 1 -- 172.21.15.107:0/878243632 learned_addr learned my addr 172.21.15.107:0/878243632 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:28.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.854+0000 7fa0c7fff640 1 --2- 172.21.15.107:0/878243632 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa0d00714f0 0x7fa0d01eda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:28.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.854+0000 7fa0d4fce640 1 --2- 172.21.15.107:0/878243632 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa0d0151180 0x7fa0d01edf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:28.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.854+0000 7fa0cffff640 1 -- 172.21.15.107:0/878243632 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa0d00714f0 msgr2=0x7fa0d01eda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:28.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.854+0000 7fa0cffff640 1 --2- 172.21.15.107:0/878243632 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa0d00714f0 0x7fa0d01eda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:28.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.854+0000 7fa0cffff640 1 -- 172.21.15.107:0/878243632 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa0d0151180 msgr2=0x7fa0d01edf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:28.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.854+0000 7fa0cffff640 1 --2- 172.21.15.107:0/878243632 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa0d0151180 0x7fa0d01edf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:28.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.854+0000 7fa0cffff640 1 -- 172.21.15.107:0/878243632 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c0009420 con 0x7fa0d0070380 2023-12-04T05:46:28.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.854+0000 7fa0d4fce640 1 --2- 172.21.15.107:0/878243632 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa0d0151180 0x7fa0d01edf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:46:28.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.854+0000 7fa0cffff640 1 --2- 172.21.15.107:0/878243632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0d0070380 0x7fa0d00ca6a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa0c00b3040 tx=0x7fa0c001b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:28.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.854+0000 7fa0cd7fa640 1 -- 172.21.15.107:0/878243632 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c00b9630 con 0x7fa0d0070380 2023-12-04T05:46:28.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.854+0000 7fa0cd7fa640 1 -- 172.21.15.107:0/878243632 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c00b97b0 con 0x7fa0d0070380 2023-12-04T05:46:28.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.854+0000 7fa0cd7fa640 1 -- 172.21.15.107:0/878243632 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c00b9b60 con 0x7fa0d0070380 2023-12-04T05:46:28.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.854+0000 7fa0d6a31640 1 -- 172.21.15.107:0/878243632 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0d014cb20 con 0x7fa0d0070380 2023-12-04T05:46:28.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.854+0000 7fa0d6a31640 1 -- 172.21.15.107:0/878243632 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0d01f1140 con 0x7fa0d0070380 2023-12-04T05:46:28.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.854+0000 7fa0d6a31640 1 -- 172.21.15.107:0/878243632 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0d01f1d50 con 0x7fa0d0070380 2023-12-04T05:46:28.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.854+0000 7fa0cd7fa640 1 -- 172.21.15.107:0/878243632 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa0c00b7040 con 0x7fa0d0070380 2023-12-04T05:46:28.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.854+0000 7fa0cd7fa640 1 --2- 172.21.15.107:0/878243632 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa0b00668d0 0x7fa0b0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:28.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.854+0000 7fa0cd7fa640 1 -- 172.21.15.107:0/878243632 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fa0c0122570 con 0x7fa0d0070380 2023-12-04T05:46:28.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.854+0000 7fa0c7fff640 1 --2- 172.21.15.107:0/878243632 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa0b00668d0 0x7fa0b0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:28.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.854+0000 7fa0c7fff640 1 --2- 172.21.15.107:0/878243632 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa0b00668d0 0x7fa0b0068d90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0147400 tx=0x7fa0b8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:28.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:28.858+0000 7fa0cd7fa640 1 -- 172.21.15.107:0/878243632 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa0c00ef550 con 0x7fa0d0070380 2023-12-04T05:46:29.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:29.050+0000 7fa0d6a31640 1 -- 172.21.15.107:0/878243632 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0d0067870 con 0x7fa0d0070380 2023-12-04T05:46:29.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:29.050+0000 7fa0cd7fa640 1 -- 172.21.15.107:0/878243632 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7fa0c00f2b70 con 0x7fa0d0070380 2023-12-04T05:46:29.053 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:29.053 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:29.053 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:46:29.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:29.054+0000 7fa0d6a31640 1 -- 172.21.15.107:0/878243632 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa0b00668d0 msgr2=0x7fa0b0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:29.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:29.054+0000 7fa0d6a31640 1 --2- 172.21.15.107:0/878243632 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa0b00668d0 0x7fa0b0068d90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0147400 tx=0x7fa0b8008040 comp rx=0 tx=0).stop 2023-12-04T05:46:29.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:29.054+0000 7fa0d6a31640 1 -- 172.21.15.107:0/878243632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0d0070380 msgr2=0x7fa0d00ca6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:29.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:29.054+0000 7fa0d6a31640 1 --2- 172.21.15.107:0/878243632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0d0070380 0x7fa0d00ca6a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa0c00b3040 tx=0x7fa0c001b9f0 comp rx=0 tx=0).stop 2023-12-04T05:46:29.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:29.054+0000 7fa0d6a31640 1 -- 172.21.15.107:0/878243632 shutdown_connections 2023-12-04T05:46:29.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:29.054+0000 7fa0d6a31640 1 --2- 172.21.15.107:0/878243632 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa0b00668d0 0x7fa0b0068d90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:29.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:29.054+0000 7fa0d6a31640 1 --2- 172.21.15.107:0/878243632 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa0d0151180 0x7fa0d01edf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:29.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:29.054+0000 7fa0d6a31640 1 --2- 172.21.15.107:0/878243632 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa0d00714f0 0x7fa0d01eda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:29.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:29.054+0000 7fa0d6a31640 1 --2- 172.21.15.107:0/878243632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0d0070380 0x7fa0d00ca6a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:29.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:29.054+0000 7fa0d6a31640 1 -- 172.21.15.107:0/878243632 >> 172.21.15.107:0/878243632 conn(0x7fa0d00c7ff0 msgr2=0x7fa0d014f080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:29.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:29.054+0000 7fa0d6a31640 1 -- 172.21.15.107:0/878243632 shutdown_connections 2023-12-04T05:46:29.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:29.054+0000 7fa0d6a31640 1 -- 172.21.15.107:0/878243632 wait complete. 2023-12-04T05:46:29.074 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 50, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-12-04T05:39:59.032019+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_4288": { "addr": "172.21.15.195:6837/3412865477", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6835", "nonce": 3412865477, "type": "v2" }, { "addr": "172.21.15.195:6837", "nonce": 3412865477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_5190": { "addr": "172.21.15.107:6836/2928623624", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 2928623624, "type": "v2" }, { "addr": "172.21.15.107:6836", "nonce": 2928623624, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5190, "incarnation": 45, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:rejoin", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-12-04T05:46:11.040247+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2, 3, 4 ], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 5190 } } } ], "standbys": [ { "addr": "172.21.15.107:6837/2752497349", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6835", "nonce": 2752497349, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 2752497349, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 49, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5292, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-12-04T05:46:30.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:46:30.940 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:46:31.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:31.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.202+0000 7ffbb1b62640 1 Processor -- start 2023-12-04T05:46:31.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.202+0000 7ffbb1b62640 1 -- start start 2023-12-04T05:46:31.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.206+0000 7ffbb1b62640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffbac154020 0x7ffbac15a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:31.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.206+0000 7ffbb1b62640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbac15ad50 con 0x7ffbac0d01c0 2023-12-04T05:46:31.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.206+0000 7ffbb1b62640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbac15d710 con 0x7ffbac0c4c10 2023-12-04T05:46:31.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.206+0000 7ffbb1b62640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbac0ca230 con 0x7ffbac154020 2023-12-04T05:46:31.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.206+0000 7ffbaaffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ffbac0d01c0 0x7ffbac071ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33520/0 (socket says 172.21.15.107:33520) 2023-12-04T05:46:31.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.206+0000 7ffbaaffd640 1 -- 172.21.15.107:0/137198780 learned_addr learned my addr 172.21.15.107:0/137198780 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:46:31.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.206+0000 7ffbab7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffbac154020 0x7ffbac15a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:31.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.206+0000 7ffbab7fe640 1 -- 172.21.15.107:0/137198780 >> v1:172.21.15.195:6789/0 conn(0x7ffbac0c4c10 legacy=0x7ffbac0c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:31.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.206+0000 7ffbab7fe640 1 -- 172.21.15.107:0/137198780 >> v1:172.21.15.107:6789/0 conn(0x7ffbac0d01c0 legacy=0x7ffbac071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:31.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.206+0000 7ffbab7fe640 1 -- 172.21.15.107:0/137198780 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbac15d9f0 con 0x7ffbac154020 2023-12-04T05:46:31.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.206+0000 7ffbab7fe640 1 --2- 172.21.15.107:0/137198780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbac154020 0x7ffbac15a810 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c0097c0 tx=0x7ffb9c01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a954eb24e6e2a294 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:31.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.206+0000 7ffbaa7fc640 1 -- 172.21.15.107:0/137198780 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c0ab600 con 0x7ffbac154020 2023-12-04T05:46:31.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.206+0000 7ffbaa7fc640 1 -- 172.21.15.107:0/137198780 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb9c0a7050 con 0x7ffbac154020 2023-12-04T05:46:31.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.206+0000 7ffbaa7fc640 1 -- 172.21.15.107:0/137198780 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c0ab980 con 0x7ffbac154020 2023-12-04T05:46:31.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbb1b62640 1 -- 172.21.15.107:0/137198780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbac154020 msgr2=0x7ffbac15a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:31.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbb1b62640 1 --2- 172.21.15.107:0/137198780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbac154020 0x7ffbac15a810 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c0097c0 tx=0x7ffb9c01c440 comp rx=0 tx=0).stop 2023-12-04T05:46:31.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbb1b62640 1 -- 172.21.15.107:0/137198780 shutdown_connections 2023-12-04T05:46:31.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbb1b62640 1 --2- 172.21.15.107:0/137198780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbac154020 0x7ffbac15a810 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:31.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbb1b62640 1 -- 172.21.15.107:0/137198780 >> 172.21.15.107:0/137198780 conn(0x7ffbac06c6c0 msgr2=0x7ffbac0cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:31.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbb1b62640 1 -- 172.21.15.107:0/137198780 shutdown_connections 2023-12-04T05:46:31.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbb1b62640 1 -- 172.21.15.107:0/137198780 wait complete. 2023-12-04T05:46:31.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbb1b62640 1 Processor -- start 2023-12-04T05:46:31.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbb1b62640 1 -- start start 2023-12-04T05:46:31.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbb1b62640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffbac0c4c10 0x7ffbac15ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:31.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbb1b62640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffbac0d01c0 0x7ffbac1f5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:31.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbb1b62640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbac154020 0x7ffbac1f6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:31.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbb1b62640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac0cc970 con 0x7ffbac154020 2023-12-04T05:46:31.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbb1b62640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac0b4b10 con 0x7ffbac0c4c10 2023-12-04T05:46:31.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbb1b62640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac0b3f70 con 0x7ffbac0d01c0 2023-12-04T05:46:31.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbabfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbac154020 0x7ffbac1f6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:31.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbabfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbac154020 0x7ffbac1f6530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55072/0 (socket says 172.21.15.107:55072) 2023-12-04T05:46:31.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbabfff640 1 -- 172.21.15.107:0/1740776581 learned_addr learned my addr 172.21.15.107:0/1740776581 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:31.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbab7fe640 1 --2- 172.21.15.107:0/1740776581 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffbac0c4c10 0x7ffbac15ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:31.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbaaffd640 1 --2- 172.21.15.107:0/1740776581 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffbac0d01c0 0x7ffbac1f5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:31.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbabfff640 1 -- 172.21.15.107:0/1740776581 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffbac0d01c0 msgr2=0x7ffbac1f5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:31.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbabfff640 1 --2- 172.21.15.107:0/1740776581 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffbac0d01c0 0x7ffbac1f5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:31.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbabfff640 1 -- 172.21.15.107:0/1740776581 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffbac0c4c10 msgr2=0x7ffbac15ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:31.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbabfff640 1 --2- 172.21.15.107:0/1740776581 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffbac0c4c10 0x7ffbac15ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:31.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbabfff640 1 -- 172.21.15.107:0/1740776581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9c009420 con 0x7ffbac154020 2023-12-04T05:46:31.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbab7fe640 1 --2- 172.21.15.107:0/1740776581 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffbac0c4c10 0x7ffbac15ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:46:31.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbabfff640 1 --2- 172.21.15.107:0/1740776581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbac154020 0x7ffbac1f6530 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ffba400c7e0 tx=0x7ffba400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:31.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffb93fff640 1 -- 172.21.15.107:0/1740776581 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba4004b70 con 0x7ffbac154020 2023-12-04T05:46:31.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffb93fff640 1 -- 172.21.15.107:0/1740776581 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba4004cd0 con 0x7ffbac154020 2023-12-04T05:46:31.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffb93fff640 1 -- 172.21.15.107:0/1740776581 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba400aa90 con 0x7ffbac154020 2023-12-04T05:46:31.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbb1b62640 1 -- 172.21.15.107:0/1740776581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbac0b41e0 con 0x7ffbac154020 2023-12-04T05:46:31.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.210+0000 7ffbb1b62640 1 -- 172.21.15.107:0/1740776581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbac0b47a0 con 0x7ffbac154020 2023-12-04T05:46:31.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.214+0000 7ffbb1b62640 1 -- 172.21.15.107:0/1740776581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbac067870 con 0x7ffbac154020 2023-12-04T05:46:31.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.214+0000 7ffb93fff640 1 -- 172.21.15.107:0/1740776581 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ffba400abf0 con 0x7ffbac154020 2023-12-04T05:46:31.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.214+0000 7ffb93fff640 1 --2- 172.21.15.107:0/1740776581 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ffb980668d0 0x7ffb98068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:31.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.214+0000 7ffb93fff640 1 -- 172.21.15.107:0/1740776581 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7ffba4084930 con 0x7ffbac154020 2023-12-04T05:46:31.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.214+0000 7ffbab7fe640 1 --2- 172.21.15.107:0/1740776581 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ffb980668d0 0x7ffb98068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:31.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.214+0000 7ffbab7fe640 1 --2- 172.21.15.107:0/1740776581 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ffb980668d0 0x7ffb98068d90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c0ad000 tx=0x7ffb9c0a6000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:31.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.218+0000 7ffb93fff640 1 -- 172.21.15.107:0/1740776581 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ffba4051870 con 0x7ffbac154020 2023-12-04T05:46:31.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.398+0000 7ffbb1b62640 1 -- 172.21.15.107:0/1740776581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbac1f5d50 con 0x7ffbac154020 2023-12-04T05:46:31.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.398+0000 7ffb93fff640 1 -- 172.21.15.107:0/1740776581 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7ffba4054e90 con 0x7ffbac154020 2023-12-04T05:46:31.402 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:31.402 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:31.403 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:46:31.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.402+0000 7ffbb1b62640 1 -- 172.21.15.107:0/1740776581 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ffb980668d0 msgr2=0x7ffb98068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:31.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.402+0000 7ffbb1b62640 1 --2- 172.21.15.107:0/1740776581 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ffb980668d0 0x7ffb98068d90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c0ad000 tx=0x7ffb9c0a6000 comp rx=0 tx=0).stop 2023-12-04T05:46:31.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.402+0000 7ffbb1b62640 1 -- 172.21.15.107:0/1740776581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbac154020 msgr2=0x7ffbac1f6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:31.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.402+0000 7ffbb1b62640 1 --2- 172.21.15.107:0/1740776581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbac154020 0x7ffbac1f6530 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ffba400c7e0 tx=0x7ffba400cca0 comp rx=0 tx=0).stop 2023-12-04T05:46:31.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.402+0000 7ffbb1b62640 1 -- 172.21.15.107:0/1740776581 shutdown_connections 2023-12-04T05:46:31.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.402+0000 7ffbb1b62640 1 --2- 172.21.15.107:0/1740776581 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ffb980668d0 0x7ffb98068d90 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:31.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.402+0000 7ffbb1b62640 1 --2- 172.21.15.107:0/1740776581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbac154020 0x7ffbac1f6530 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:31.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.402+0000 7ffbb1b62640 1 --2- 172.21.15.107:0/1740776581 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ffbac0d01c0 0x7ffbac1f5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:31.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.402+0000 7ffbb1b62640 1 --2- 172.21.15.107:0/1740776581 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffbac0c4c10 0x7ffbac15ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:31.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.402+0000 7ffbb1b62640 1 -- 172.21.15.107:0/1740776581 >> 172.21.15.107:0/1740776581 conn(0x7ffbac06c6c0 msgr2=0x7ffbac0ce980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:31.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.402+0000 7ffbb1b62640 1 -- 172.21.15.107:0/1740776581 shutdown_connections 2023-12-04T05:46:31.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:31.402+0000 7ffbb1b62640 1 -- 172.21.15.107:0/1740776581 wait complete. 2023-12-04T05:46:31.419 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:46:33.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:33.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.550+0000 7fd27d9bc640 1 Processor -- start 2023-12-04T05:46:33.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.550+0000 7fd27d9bc640 1 -- start start 2023-12-04T05:46:33.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.550+0000 7fd27d9bc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd2780ca6b0 0x7fd2780cd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:33.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.550+0000 7fd27d9bc640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd2780c9a00 0x7fd2780cda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:33.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.550+0000 7fd27d9bc640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2780c9570 con 0x7fd2780ca2d0 2023-12-04T05:46:33.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.550+0000 7fd27d9bc640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd278073290 con 0x7fd2780ca6b0 2023-12-04T05:46:33.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.550+0000 7fd27d9bc640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2780c8c80 con 0x7fd2780c9a00 2023-12-04T05:46:33.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.550+0000 7fd2767fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd2780ca6b0 0x7fd2780cd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:33.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.550+0000 7fd2767fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd2780ca6b0 0x7fd2780cd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55082/0 (socket says 172.21.15.107:55082) 2023-12-04T05:46:33.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.550+0000 7fd2767fc640 1 -- 172.21.15.107:0/3916715786 learned_addr learned my addr 172.21.15.107:0/3916715786 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:33.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.550+0000 7fd2777fe640 1 --2- 172.21.15.107:0/3916715786 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd2780c9a00 0x7fd2780cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:33.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.550+0000 7fd2767fc640 1 -- 172.21.15.107:0/3916715786 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd2780c9a00 msgr2=0x7fd2780cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:33.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.550+0000 7fd2767fc640 1 --2- 172.21.15.107:0/3916715786 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd2780c9a00 0x7fd2780cda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:33.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.550+0000 7fd2767fc640 1 -- 172.21.15.107:0/3916715786 >> v1:172.21.15.195:6789/0 conn(0x7fd2780ca2d0 legacy=0x7fd278072570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:33.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.550+0000 7fd2767fc640 1 -- 172.21.15.107:0/3916715786 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd278072fb0 con 0x7fd2780ca6b0 2023-12-04T05:46:33.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd2767fc640 1 --2- 172.21.15.107:0/3916715786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2780ca6b0 0x7fd2780cd510 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd2600098a0 tx=0x7fd26001ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=477e06daa7f11a97 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:33.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd275ffb640 1 -- 172.21.15.107:0/3916715786 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2600a8990 con 0x7fd2780ca6b0 2023-12-04T05:46:33.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd275ffb640 1 -- 172.21.15.107:0/3916715786 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2600a8af0 con 0x7fd2780ca6b0 2023-12-04T05:46:33.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd275ffb640 1 -- 172.21.15.107:0/3916715786 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2600a8e80 con 0x7fd2780ca6b0 2023-12-04T05:46:33.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd27d9bc640 1 -- 172.21.15.107:0/3916715786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2780ca6b0 msgr2=0x7fd2780cd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:33.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd27d9bc640 1 --2- 172.21.15.107:0/3916715786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2780ca6b0 0x7fd2780cd510 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd2600098a0 tx=0x7fd26001ecd0 comp rx=0 tx=0).stop 2023-12-04T05:46:33.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd27d9bc640 1 -- 172.21.15.107:0/3916715786 shutdown_connections 2023-12-04T05:46:33.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd27d9bc640 1 --2- 172.21.15.107:0/3916715786 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd2780c9a00 0x7fd2780cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:33.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd27d9bc640 1 --2- 172.21.15.107:0/3916715786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2780ca6b0 0x7fd2780cd510 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:33.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd27d9bc640 1 -- 172.21.15.107:0/3916715786 >> 172.21.15.107:0/3916715786 conn(0x7fd2780663a0 msgr2=0x7fd2780667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:33.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd27d9bc640 1 -- 172.21.15.107:0/3916715786 shutdown_connections 2023-12-04T05:46:33.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd27d9bc640 1 -- 172.21.15.107:0/3916715786 wait complete. 2023-12-04T05:46:33.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd27d9bc640 1 Processor -- start 2023-12-04T05:46:33.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd27d9bc640 1 -- start start 2023-12-04T05:46:33.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd27d9bc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2780c9a00 0x7fd27806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:33.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd27d9bc640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2780ca2d0 0x7fd2781406f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:33.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd27d9bc640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd27813ead0 0x7fd278140c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:33.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd27d9bc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2781eb620 con 0x7fd2780c9a00 2023-12-04T05:46:33.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd27d9bc640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2781eb900 con 0x7fd2780ca2d0 2023-12-04T05:46:33.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd27d9bc640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2781ebbe0 con 0x7fd27813ead0 2023-12-04T05:46:33.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd276ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2780c9a00 0x7fd27806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:33.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd276ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2780c9a00 0x7fd27806cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55090/0 (socket says 172.21.15.107:55090) 2023-12-04T05:46:33.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd276ffd640 1 -- 172.21.15.107:0/772318764 learned_addr learned my addr 172.21.15.107:0/772318764 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:33.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd2777fe640 1 --2- 172.21.15.107:0/772318764 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd27813ead0 0x7fd278140c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:33.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd2767fc640 1 --2- 172.21.15.107:0/772318764 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2780ca2d0 0x7fd2781406f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:33.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd276ffd640 1 -- 172.21.15.107:0/772318764 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd27813ead0 msgr2=0x7fd278140c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:33.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd276ffd640 1 --2- 172.21.15.107:0/772318764 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd27813ead0 0x7fd278140c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:33.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd276ffd640 1 -- 172.21.15.107:0/772318764 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2780ca2d0 msgr2=0x7fd2781406f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:33.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd276ffd640 1 --2- 172.21.15.107:0/772318764 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2780ca2d0 0x7fd2781406f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:33.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.554+0000 7fd276ffd640 1 -- 172.21.15.107:0/772318764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2600094d0 con 0x7fd2780c9a00 2023-12-04T05:46:33.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.558+0000 7fd276ffd640 1 --2- 172.21.15.107:0/772318764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2780c9a00 0x7fd27806cc90 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd26800c7c0 tx=0x7fd26800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:33.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.558+0000 7fd26ffff640 1 -- 172.21.15.107:0/772318764 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26800aa50 con 0x7fd2780c9a00 2023-12-04T05:46:33.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.558+0000 7fd26ffff640 1 -- 172.21.15.107:0/772318764 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26800abb0 con 0x7fd2780c9a00 2023-12-04T05:46:33.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.558+0000 7fd27d9bc640 1 -- 172.21.15.107:0/772318764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2781ebec0 con 0x7fd2780c9a00 2023-12-04T05:46:33.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.558+0000 7fd26ffff640 1 -- 172.21.15.107:0/772318764 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd268012420 con 0x7fd2780c9a00 2023-12-04T05:46:33.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.558+0000 7fd27d9bc640 1 -- 172.21.15.107:0/772318764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2781eb310 con 0x7fd2780c9a00 2023-12-04T05:46:33.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.558+0000 7fd27d9bc640 1 -- 172.21.15.107:0/772318764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd27806b6a0 con 0x7fd2780c9a00 2023-12-04T05:46:33.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.558+0000 7fd26ffff640 1 -- 172.21.15.107:0/772318764 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fd268012580 con 0x7fd2780c9a00 2023-12-04T05:46:33.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.558+0000 7fd26ffff640 1 --2- 172.21.15.107:0/772318764 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fd2580668d0 0x7fd258068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:33.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.558+0000 7fd26ffff640 1 -- 172.21.15.107:0/772318764 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fd2680848d0 con 0x7fd2780c9a00 2023-12-04T05:46:33.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.558+0000 7fd2767fc640 1 --2- 172.21.15.107:0/772318764 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fd2580668d0 0x7fd258068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:33.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.558+0000 7fd2767fc640 1 --2- 172.21.15.107:0/772318764 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fd2580668d0 0x7fd258068d90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd2600b2000 tx=0x7fd260008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:33.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.562+0000 7fd26ffff640 1 -- 172.21.15.107:0/772318764 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd268051940 con 0x7fd2780c9a00 2023-12-04T05:46:33.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.750+0000 7fd27d9bc640 1 -- 172.21.15.107:0/772318764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd278067ec0 con 0x7fd2780c9a00 2023-12-04T05:46:33.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.750+0000 7fd26ffff640 1 -- 172.21.15.107:0/772318764 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7fd268016090 con 0x7fd2780c9a00 2023-12-04T05:46:33.754 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:33.754 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:46:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.754+0000 7fd27d9bc640 1 -- 172.21.15.107:0/772318764 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fd2580668d0 msgr2=0x7fd258068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.754+0000 7fd27d9bc640 1 --2- 172.21.15.107:0/772318764 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fd2580668d0 0x7fd258068d90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd2600b2000 tx=0x7fd260008040 comp rx=0 tx=0).stop 2023-12-04T05:46:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.754+0000 7fd27d9bc640 1 -- 172.21.15.107:0/772318764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2780c9a00 msgr2=0x7fd27806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.754+0000 7fd27d9bc640 1 --2- 172.21.15.107:0/772318764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2780c9a00 0x7fd27806cc90 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd26800c7c0 tx=0x7fd26800cc80 comp rx=0 tx=0).stop 2023-12-04T05:46:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.754+0000 7fd27d9bc640 1 -- 172.21.15.107:0/772318764 shutdown_connections 2023-12-04T05:46:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.754+0000 7fd27d9bc640 1 --2- 172.21.15.107:0/772318764 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fd2580668d0 0x7fd258068d90 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.754+0000 7fd27d9bc640 1 --2- 172.21.15.107:0/772318764 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd27813ead0 0x7fd278140c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.754+0000 7fd27d9bc640 1 --2- 172.21.15.107:0/772318764 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2780ca2d0 0x7fd2781406f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.754+0000 7fd27d9bc640 1 --2- 172.21.15.107:0/772318764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2780c9a00 0x7fd27806cc90 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.754+0000 7fd27d9bc640 1 -- 172.21.15.107:0/772318764 >> 172.21.15.107:0/772318764 conn(0x7fd2780663a0 msgr2=0x7fd278066780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.754+0000 7fd27d9bc640 1 -- 172.21.15.107:0/772318764 shutdown_connections 2023-12-04T05:46:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:33.754+0000 7fd27d9bc640 1 -- 172.21.15.107:0/772318764 wait complete. 2023-12-04T05:46:33.773 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:46:34.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~53s 2023-12-04T05:46:34.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~167s 2023-12-04T05:46:34.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~208s 2023-12-04T05:46:35.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:35.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f7091eee640 1 Processor -- start 2023-12-04T05:46:35.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f7091eee640 1 -- start start 2023-12-04T05:46:35.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f7091eee640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f708c0ca2d0 0x7f708c072570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:35.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f7091eee640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f708c072ab0 0x7f708c151960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:35.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f7091eee640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f708c0c9570 con 0x7f708c0ca6b0 2023-12-04T05:46:35.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f7091eee640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f708c073290 con 0x7f708c072ab0 2023-12-04T05:46:35.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f7091eee640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f708c0c8c80 con 0x7f708c0ca2d0 2023-12-04T05:46:35.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f7082ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f708c0ca6b0 0x7f708c0cd510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33536/0 (socket says 172.21.15.107:33536) 2023-12-04T05:46:35.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f7082ffd640 1 -- 172.21.15.107:0/1407494173 learned_addr learned my addr 172.21.15.107:0/1407494173 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:46:35.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708bfff640 1 --2- 172.21.15.107:0/1407494173 >> v2:172.21.15.195:3300/0 conn(0x7f708c072ab0 0x7f708c151960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:35.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708b7fe640 1 --2- 172.21.15.107:0/1407494173 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f708c0ca2d0 0x7f708c072570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:35.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708affd640 1 -- 172.21.15.107:0/1407494173 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3933858943 0 0) 0x7f708c0c9570 con 0x7f708c0ca6b0 2023-12-04T05:46:35.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708affd640 1 -- 172.21.15.107:0/1407494173 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7078003530 con 0x7f708c0ca6b0 2023-12-04T05:46:35.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708affd640 1 -- 172.21.15.107:0/1407494173 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 208467781 0 0) 0x7f7078003530 con 0x7f708c0ca6b0 2023-12-04T05:46:35.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708affd640 1 -- 172.21.15.107:0/1407494173 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f708c0ca2d0 msgr2=0x7f708c072570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:35.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708affd640 1 --2- 172.21.15.107:0/1407494173 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f708c0ca2d0 0x7f708c072570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:35.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708affd640 1 -- 172.21.15.107:0/1407494173 >> v2:172.21.15.195:3300/0 conn(0x7f708c072ab0 msgr2=0x7f708c151960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:35.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708affd640 1 --2- 172.21.15.107:0/1407494173 >> v2:172.21.15.195:3300/0 conn(0x7f708c072ab0 0x7f708c151960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:35.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708affd640 1 -- 172.21.15.107:0/1407494173 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f708c072fb0 con 0x7f708c0ca6b0 2023-12-04T05:46:35.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708affd640 1 -- 172.21.15.107:0/1407494173 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f7074001680 con 0x7f708c0ca6b0 2023-12-04T05:46:35.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708affd640 1 -- 172.21.15.107:0/1407494173 >> v1:172.21.15.107:6789/0 conn(0x7f708c0ca6b0 legacy=0x7f708c0cd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:35.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708affd640 1 --2- 172.21.15.107:0/1407494173 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f708c0ca2d0 0x7f7078003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:35.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708affd640 1 --2- 172.21.15.107:0/1407494173 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7078003f00 0x7f70780047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:35.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708affd640 1 --2- 172.21.15.107:0/1407494173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7078004d10 0x7f7078005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:35.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708affd640 1 -- 172.21.15.107:0/1407494173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c073290 con 0x7f7078004d10 2023-12-04T05:46:35.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708affd640 1 -- 172.21.15.107:0/1407494173 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c0c8c80 con 0x7f7078003f00 2023-12-04T05:46:35.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708affd640 1 -- 172.21.15.107:0/1407494173 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f707800b9c0 con 0x7f708c0ca2d0 2023-12-04T05:46:35.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708bfff640 1 --2- 172.21.15.107:0/1407494173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7078004d10 0x7f7078005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:35.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f7082ffd640 1 --2- 172.21.15.107:0/1407494173 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7078003f00 0x7f70780047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:35.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708b7fe640 1 --2- 172.21.15.107:0/1407494173 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f708c0ca2d0 0x7f7078003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:35.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708bfff640 1 -- 172.21.15.107:0/1407494173 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f708c0ca2d0 msgr2=0x7f7078003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:35.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708bfff640 1 --2- 172.21.15.107:0/1407494173 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f708c0ca2d0 0x7f7078003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:35.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708bfff640 1 -- 172.21.15.107:0/1407494173 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7078003f00 msgr2=0x7f70780047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:35.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708bfff640 1 --2- 172.21.15.107:0/1407494173 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7078003f00 0x7f70780047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:35.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708bfff640 1 -- 172.21.15.107:0/1407494173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7078003530 con 0x7f7078004d10 2023-12-04T05:46:35.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f708b7fe640 1 --2- 172.21.15.107:0/1407494173 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f708c0ca2d0 0x7f7078003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:46:35.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.902+0000 7f7082ffd640 1 --2- 172.21.15.107:0/1407494173 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7078003f00 0x7f70780047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:46:35.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.906+0000 7f708bfff640 1 --2- 172.21.15.107:0/1407494173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7078004d10 0x7f7078005190 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7084009fc0 tx=0x7f708401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25813582f5bc2aa3 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:35.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.906+0000 7f708affd640 1 -- 172.21.15.107:0/1407494173 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70840a5cb0 con 0x7f7078004d10 2023-12-04T05:46:35.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.906+0000 7f708affd640 1 -- 172.21.15.107:0/1407494173 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f708400f040 con 0x7f7078004d10 2023-12-04T05:46:35.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.906+0000 7f708affd640 1 -- 172.21.15.107:0/1407494173 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70840ad680 con 0x7f7078004d10 2023-12-04T05:46:35.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.906+0000 7f7091eee640 1 -- 172.21.15.107:0/1407494173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7078004d10 msgr2=0x7f7078005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:35.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.906+0000 7f7091eee640 1 --2- 172.21.15.107:0/1407494173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7078004d10 0x7f7078005190 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7084009fc0 tx=0x7f708401cca0 comp rx=0 tx=0).stop 2023-12-04T05:46:35.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.906+0000 7f708bfff640 1 -- 172.21.15.107:0/1407494173 reap_dead start 2023-12-04T05:46:35.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f7091eee640 1 -- 172.21.15.107:0/1407494173 shutdown_connections 2023-12-04T05:46:35.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f7091eee640 1 -- 172.21.15.107:0/1407494173 >> 172.21.15.107:0/1407494173 conn(0x7f708c0663a0 msgr2=0x7f708c0667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:35.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f7091eee640 1 -- 172.21.15.107:0/1407494173 shutdown_connections 2023-12-04T05:46:35.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f7091eee640 1 -- 172.21.15.107:0/1407494173 wait complete. 2023-12-04T05:46:35.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f7091eee640 1 Processor -- start 2023-12-04T05:46:35.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f7091eee640 1 -- start start 2023-12-04T05:46:35.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f7091eee640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7078004d10 0x7f708c141ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:35.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f7091eee640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f708c156690 0x7f708c146370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:35.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f7091eee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f708c145b60 0x7f708c1468b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:35.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f7091eee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c157980 con 0x7f708c145b60 2023-12-04T05:46:35.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f7091eee640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c1436a0 con 0x7f708c156690 2023-12-04T05:46:35.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f7091eee640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f708c1426c0 con 0x7f7078004d10 2023-12-04T05:46:35.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f708bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f708c145b60 0x7f708c1468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:35.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f708bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f708c145b60 0x7f708c1468b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55104/0 (socket says 172.21.15.107:55104) 2023-12-04T05:46:35.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f708bfff640 1 -- 172.21.15.107:0/1215660855 learned_addr learned my addr 172.21.15.107:0/1215660855 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:35.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f7082ffd640 1 --2- 172.21.15.107:0/1215660855 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f708c156690 0x7f708c146370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:35.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f708b7fe640 1 --2- 172.21.15.107:0/1215660855 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7078004d10 0x7f708c141ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:35.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f708bfff640 1 -- 172.21.15.107:0/1215660855 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7078004d10 msgr2=0x7f708c141ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:35.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f708bfff640 1 --2- 172.21.15.107:0/1215660855 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7078004d10 0x7f708c141ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:35.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f708bfff640 1 -- 172.21.15.107:0/1215660855 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f708c156690 msgr2=0x7f708c146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:35.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f708bfff640 1 --2- 172.21.15.107:0/1215660855 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f708c156690 0x7f708c146370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:35.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f708bfff640 1 -- 172.21.15.107:0/1215660855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f708c142a90 con 0x7f708c145b60 2023-12-04T05:46:35.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f7082ffd640 1 --2- 172.21.15.107:0/1215660855 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f708c156690 0x7f708c146370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:46:35.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f708bfff640 1 --2- 172.21.15.107:0/1215660855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f708c145b60 0x7f708c1468b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f7084002240 tx=0x7f7084009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:35.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f7088ff9640 1 -- 172.21.15.107:0/1215660855 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70840039a0 con 0x7f708c145b60 2023-12-04T05:46:35.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f7088ff9640 1 -- 172.21.15.107:0/1215660855 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7084003b00 con 0x7f708c145b60 2023-12-04T05:46:35.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f7091eee640 1 -- 172.21.15.107:0/1215660855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f708c142d70 con 0x7f708c145b60 2023-12-04T05:46:35.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f7088ff9640 1 -- 172.21.15.107:0/1215660855 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7084003e70 con 0x7f708c145b60 2023-12-04T05:46:35.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f7091eee640 1 -- 172.21.15.107:0/1215660855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f708c143330 con 0x7f708c145b60 2023-12-04T05:46:35.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.910+0000 7f7091eee640 1 -- 172.21.15.107:0/1215660855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f708c143810 con 0x7f708c145b60 2023-12-04T05:46:35.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.914+0000 7f7088ff9640 1 -- 172.21.15.107:0/1215660855 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f70840b4040 con 0x7f708c145b60 2023-12-04T05:46:35.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.914+0000 7f7088ff9640 1 --2- 172.21.15.107:0/1215660855 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f706c066900 0x7f706c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:35.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.914+0000 7f7088ff9640 1 -- 172.21.15.107:0/1215660855 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f708411e6f0 con 0x7f708c145b60 2023-12-04T05:46:35.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.914+0000 7f708b7fe640 1 --2- 172.21.15.107:0/1215660855 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f706c066900 0x7f706c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:35.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.914+0000 7f708b7fe640 1 --2- 172.21.15.107:0/1215660855 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f706c066900 0x7f706c068dc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f707c009ac0 tx=0x7f707c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:35.918+0000 7f7088ff9640 1 -- 172.21.15.107:0/1215660855 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f70840eb6e0 con 0x7f708c145b60 2023-12-04T05:46:36.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:36.098+0000 7f7091eee640 1 -- 172.21.15.107:0/1215660855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f708c0c88b0 con 0x7f708c145b60 2023-12-04T05:46:36.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:36.098+0000 7f7088ff9640 1 -- 172.21.15.107:0/1215660855 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f70840eed00 con 0x7f708c145b60 2023-12-04T05:46:36.101 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:36.102 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:36.102 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:46:36.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:36.102+0000 7f7091eee640 1 -- 172.21.15.107:0/1215660855 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f706c066900 msgr2=0x7f706c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:36.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:36.102+0000 7f7091eee640 1 --2- 172.21.15.107:0/1215660855 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f706c066900 0x7f706c068dc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f707c009ac0 tx=0x7f707c008040 comp rx=0 tx=0).stop 2023-12-04T05:46:36.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:36.102+0000 7f7091eee640 1 -- 172.21.15.107:0/1215660855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f708c145b60 msgr2=0x7f708c1468b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:36.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:36.102+0000 7f7091eee640 1 --2- 172.21.15.107:0/1215660855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f708c145b60 0x7f708c1468b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f7084002240 tx=0x7f7084009750 comp rx=0 tx=0).stop 2023-12-04T05:46:36.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:36.102+0000 7f7091eee640 1 -- 172.21.15.107:0/1215660855 shutdown_connections 2023-12-04T05:46:36.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:36.102+0000 7f7091eee640 1 --2- 172.21.15.107:0/1215660855 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f706c066900 0x7f706c068dc0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:36.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:36.102+0000 7f7091eee640 1 --2- 172.21.15.107:0/1215660855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f708c145b60 0x7f708c1468b0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:36.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:36.102+0000 7f7091eee640 1 --2- 172.21.15.107:0/1215660855 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f708c156690 0x7f708c146370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:36.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:36.102+0000 7f7091eee640 1 --2- 172.21.15.107:0/1215660855 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f7078004d10 0x7f708c141ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:36.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:36.102+0000 7f7091eee640 1 -- 172.21.15.107:0/1215660855 >> 172.21.15.107:0/1215660855 conn(0x7f708c0663a0 msgr2=0x7f708c0c8130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:36.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:36.102+0000 7f7091eee640 1 -- 172.21.15.107:0/1215660855 shutdown_connections 2023-12-04T05:46:36.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:36.102+0000 7f7091eee640 1 -- 172.21.15.107:0/1215660855 wait complete. 2023-12-04T05:46:36.119 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:46:38.121 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:38.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.250+0000 7f9d9121d640 1 Processor -- start 2023-12-04T05:46:38.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.250+0000 7f9d9121d640 1 -- start start 2023-12-04T05:46:38.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.254+0000 7f9d9121d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d8c153fd0 0x7f9d8c0c9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:38.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.254+0000 7f9d9121d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9d8c15a0d0 0x7f9d8c15a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:38.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.254+0000 7f9d9121d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8c072e40 con 0x7f9d8c0726c0 2023-12-04T05:46:38.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.254+0000 7f9d9121d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d8c0c98c0 con 0x7f9d8c15a0d0 2023-12-04T05:46:38.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.254+0000 7f9d9121d640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c15aa50 con 0x7f9d8c153fd0 2023-12-04T05:46:38.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.254+0000 7f9d8b59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9d8c15a0d0 0x7f9d8c15a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:38.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.254+0000 7f9d8b59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9d8c15a0d0 0x7f9d8c15a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44446/0 (socket says 172.21.15.107:44446) 2023-12-04T05:46:38.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.254+0000 7f9d8b59e640 1 -- 172.21.15.107:0/1677939850 learned_addr learned my addr 172.21.15.107:0/1677939850 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:38.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.254+0000 7f9d8b59e640 1 -- 172.21.15.107:0/1677939850 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d8c153fd0 msgr2=0x7f9d8c0c9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:38.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.254+0000 7f9d8ad9d640 1 --2- 172.21.15.107:0/1677939850 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d8c153fd0 0x7f9d8c0c9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:38.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.254+0000 7f9d8b59e640 1 --2- 172.21.15.107:0/1677939850 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d8c153fd0 0x7f9d8c0c9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:38.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.254+0000 7f9d8b59e640 1 -- 172.21.15.107:0/1677939850 >> v1:172.21.15.195:6789/0 conn(0x7f9d8c0726c0 legacy=0x7f9d8c06b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:38.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.254+0000 7f9d8b59e640 1 -- 172.21.15.107:0/1677939850 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8c15d410 con 0x7f9d8c15a0d0 2023-12-04T05:46:38.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.254+0000 7f9d8b59e640 1 --2- 172.21.15.107:0/1677939850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d8c15a0d0 0x7f9d8c15a510 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f9d84009e50 tx=0x7f9d8401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53fc4c5061fc9d84 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:38.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.254+0000 7f9d89d9b640 1 -- 172.21.15.107:0/1677939850 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d840af400 con 0x7f9d8c15a0d0 2023-12-04T05:46:38.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.254+0000 7f9d89d9b640 1 -- 172.21.15.107:0/1677939850 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d840ab050 con 0x7f9d8c15a0d0 2023-12-04T05:46:38.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.254+0000 7f9d89d9b640 1 -- 172.21.15.107:0/1677939850 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d840af780 con 0x7f9d8c15a0d0 2023-12-04T05:46:38.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d9121d640 1 -- 172.21.15.107:0/1677939850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d8c15a0d0 msgr2=0x7f9d8c15a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:38.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d9121d640 1 --2- 172.21.15.107:0/1677939850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d8c15a0d0 0x7f9d8c15a510 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f9d84009e50 tx=0x7f9d8401f1c0 comp rx=0 tx=0).stop 2023-12-04T05:46:38.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d9121d640 1 -- 172.21.15.107:0/1677939850 shutdown_connections 2023-12-04T05:46:38.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d9121d640 1 --2- 172.21.15.107:0/1677939850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d8c15a0d0 0x7f9d8c15a510 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:38.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d9121d640 1 --2- 172.21.15.107:0/1677939850 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d8c153fd0 0x7f9d8c0c9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:38.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d9121d640 1 -- 172.21.15.107:0/1677939850 >> 172.21.15.107:0/1677939850 conn(0x7f9d8c0663a0 msgr2=0x7f9d8c0667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:38.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d9121d640 1 -- 172.21.15.107:0/1677939850 shutdown_connections 2023-12-04T05:46:38.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d9121d640 1 -- 172.21.15.107:0/1677939850 wait complete. 2023-12-04T05:46:38.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d9121d640 1 Processor -- start 2023-12-04T05:46:38.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d9121d640 1 -- start start 2023-12-04T05:46:38.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d9121d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d8c0726c0 0x7f9d8c0c4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:38.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d9121d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d8c153fd0 0x7f9d8c14ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:38.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d9121d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d8c15a0d0 0x7f9d8c157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:38.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d9121d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c0c6630 con 0x7f9d8c153fd0 2023-12-04T05:46:38.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d9121d640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c147ea0 con 0x7f9d8c0726c0 2023-12-04T05:46:38.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d9121d640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c0c5310 con 0x7f9d8c15a0d0 2023-12-04T05:46:38.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d8a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d8c153fd0 0x7f9d8c14ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:38.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d8a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d8c153fd0 0x7f9d8c14ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44454/0 (socket says 172.21.15.107:44454) 2023-12-04T05:46:38.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d8a59c640 1 -- 172.21.15.107:0/917152696 learned_addr learned my addr 172.21.15.107:0/917152696 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:38.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d8b59e640 1 --2- 172.21.15.107:0/917152696 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d8c15a0d0 0x7f9d8c157fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:38.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d8ad9d640 1 --2- 172.21.15.107:0/917152696 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d8c0726c0 0x7f9d8c0c4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:38.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d8a59c640 1 -- 172.21.15.107:0/917152696 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d8c15a0d0 msgr2=0x7f9d8c157fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:38.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d8a59c640 1 --2- 172.21.15.107:0/917152696 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d8c15a0d0 0x7f9d8c157fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:38.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d8a59c640 1 -- 172.21.15.107:0/917152696 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d8c0726c0 msgr2=0x7f9d8c0c4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:38.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d8a59c640 1 --2- 172.21.15.107:0/917152696 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d8c0726c0 0x7f9d8c0c4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:38.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d8a59c640 1 -- 172.21.15.107:0/917152696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d84009b30 con 0x7f9d8c153fd0 2023-12-04T05:46:38.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d8a59c640 1 --2- 172.21.15.107:0/917152696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d8c153fd0 0x7f9d8c14ba20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c00c480 tx=0x7f9d7c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:38.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d837fe640 1 -- 172.21.15.107:0/917152696 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c012830 con 0x7f9d8c153fd0 2023-12-04T05:46:38.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d837fe640 1 -- 172.21.15.107:0/917152696 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7c0129b0 con 0x7f9d8c153fd0 2023-12-04T05:46:38.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d9121d640 1 -- 172.21.15.107:0/917152696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d8c147290 con 0x7f9d8c153fd0 2023-12-04T05:46:38.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d837fe640 1 -- 172.21.15.107:0/917152696 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c012d60 con 0x7f9d8c153fd0 2023-12-04T05:46:38.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d9121d640 1 -- 172.21.15.107:0/917152696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d8c147850 con 0x7f9d8c153fd0 2023-12-04T05:46:38.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.258+0000 7f9d9121d640 1 -- 172.21.15.107:0/917152696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8c15a550 con 0x7f9d8c153fd0 2023-12-04T05:46:38.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.262+0000 7f9d837fe640 1 -- 172.21.15.107:0/917152696 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9d7c01c460 con 0x7f9d8c153fd0 2023-12-04T05:46:38.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.262+0000 7f9d837fe640 1 --2- 172.21.15.107:0/917152696 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9d6c0668d0 0x7f9d6c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:38.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.262+0000 7f9d837fe640 1 -- 172.21.15.107:0/917152696 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f9d7c0845a0 con 0x7f9d8c153fd0 2023-12-04T05:46:38.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.262+0000 7f9d8ad9d640 1 --2- 172.21.15.107:0/917152696 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9d6c0668d0 0x7f9d6c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:38.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.262+0000 7f9d8ad9d640 1 --2- 172.21.15.107:0/917152696 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9d6c0668d0 0x7f9d6c068d90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400b800 tx=0x7f9d74008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:38.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.266+0000 7f9d837fe640 1 -- 172.21.15.107:0/917152696 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9d7c051600 con 0x7f9d8c153fd0 2023-12-04T05:46:38.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.450+0000 7f9d9121d640 1 -- 172.21.15.107:0/917152696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d8c067870 con 0x7f9d8c153fd0 2023-12-04T05:46:38.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.450+0000 7f9d837fe640 1 -- 172.21.15.107:0/917152696 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f9d7c054c20 con 0x7f9d8c153fd0 2023-12-04T05:46:38.453 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:38.453 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:38.453 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:46:38.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.454+0000 7f9d9121d640 1 -- 172.21.15.107:0/917152696 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9d6c0668d0 msgr2=0x7f9d6c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:38.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.454+0000 7f9d9121d640 1 --2- 172.21.15.107:0/917152696 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9d6c0668d0 0x7f9d6c068d90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400b800 tx=0x7f9d74008040 comp rx=0 tx=0).stop 2023-12-04T05:46:38.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.454+0000 7f9d9121d640 1 -- 172.21.15.107:0/917152696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d8c153fd0 msgr2=0x7f9d8c14ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:38.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.454+0000 7f9d9121d640 1 --2- 172.21.15.107:0/917152696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d8c153fd0 0x7f9d8c14ba20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c00c480 tx=0x7f9d7c00c940 comp rx=0 tx=0).stop 2023-12-04T05:46:38.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.454+0000 7f9d9121d640 1 -- 172.21.15.107:0/917152696 shutdown_connections 2023-12-04T05:46:38.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.454+0000 7f9d9121d640 1 --2- 172.21.15.107:0/917152696 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9d6c0668d0 0x7f9d6c068d90 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:38.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.454+0000 7f9d9121d640 1 --2- 172.21.15.107:0/917152696 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d8c15a0d0 0x7f9d8c157fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:38.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.454+0000 7f9d9121d640 1 --2- 172.21.15.107:0/917152696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d8c153fd0 0x7f9d8c14ba20 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:38.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.454+0000 7f9d9121d640 1 --2- 172.21.15.107:0/917152696 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d8c0726c0 0x7f9d8c0c4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:38.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.454+0000 7f9d9121d640 1 -- 172.21.15.107:0/917152696 >> 172.21.15.107:0/917152696 conn(0x7f9d8c0663a0 msgr2=0x7f9d8c149310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:38.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.454+0000 7f9d9121d640 1 -- 172.21.15.107:0/917152696 shutdown_connections 2023-12-04T05:46:38.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:38.454+0000 7f9d9121d640 1 -- 172.21.15.107:0/917152696 wait complete. 2023-12-04T05:46:38.471 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:46:40.472 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:40.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.598+0000 7f7866d64640 1 Processor -- start 2023-12-04T05:46:40.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.598+0000 7f7866d64640 1 -- start start 2023-12-04T05:46:40.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.602+0000 7f7866d64640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7860072110 0x7f7860158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:40.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.602+0000 7f7866d64640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f78600c56f0 0x7f7860140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:40.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.602+0000 7f7866d64640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7860158da0 con 0x7f78600724f0 2023-12-04T05:46:40.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.602+0000 7f7866d64640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78601583d0 con 0x7f7860072110 2023-12-04T05:46:40.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.602+0000 7f7866d64640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78600c6170 con 0x7f78600c56f0 2023-12-04T05:46:40.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.602+0000 7f7864b00640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7860072110 0x7f7860158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:40.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.602+0000 7f7864b00640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7860072110 0x7f7860158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44462/0 (socket says 172.21.15.107:44462) 2023-12-04T05:46:40.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.602+0000 7f7864b00640 1 -- 172.21.15.107:0/3378532475 learned_addr learned my addr 172.21.15.107:0/3378532475 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:40.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.602+0000 7f7865301640 1 --2- 172.21.15.107:0/3378532475 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f78600c56f0 0x7f7860140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:40.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.602+0000 7f7864b00640 1 -- 172.21.15.107:0/3378532475 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f78600c56f0 msgr2=0x7f7860140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:40.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.602+0000 7f7864b00640 1 --2- 172.21.15.107:0/3378532475 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f78600c56f0 0x7f7860140830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:40.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.602+0000 7f7864b00640 1 -- 172.21.15.107:0/3378532475 >> v1:172.21.15.195:6789/0 conn(0x7f78600724f0 legacy=0x7f78600c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:40.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.602+0000 7f7864b00640 1 -- 172.21.15.107:0/3378532475 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7860157da0 con 0x7f7860072110 2023-12-04T05:46:40.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.602+0000 7f7864b00640 1 --2- 172.21.15.107:0/3378532475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7860072110 0x7f7860158860 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f786006b8c0 tx=0x7f785001c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16bb09ec48253cd5 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:40.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.602+0000 7f785f7fe640 1 -- 172.21.15.107:0/3378532475 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78500ad640 con 0x7f7860072110 2023-12-04T05:46:40.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f785f7fe640 1 -- 172.21.15.107:0/3378532475 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78500ad7a0 con 0x7f7860072110 2023-12-04T05:46:40.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f785f7fe640 1 -- 172.21.15.107:0/3378532475 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78500adb30 con 0x7f7860072110 2023-12-04T05:46:40.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f7866d64640 1 -- 172.21.15.107:0/3378532475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7860072110 msgr2=0x7f7860158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:40.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f7866d64640 1 --2- 172.21.15.107:0/3378532475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7860072110 0x7f7860158860 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f786006b8c0 tx=0x7f785001c4e0 comp rx=0 tx=0).stop 2023-12-04T05:46:40.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f7866d64640 1 -- 172.21.15.107:0/3378532475 shutdown_connections 2023-12-04T05:46:40.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f7866d64640 1 --2- 172.21.15.107:0/3378532475 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f78600c56f0 0x7f7860140830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:40.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f7866d64640 1 --2- 172.21.15.107:0/3378532475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7860072110 0x7f7860158860 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:40.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f7866d64640 1 -- 172.21.15.107:0/3378532475 >> 172.21.15.107:0/3378532475 conn(0x7f78600c7ff0 msgr2=0x7f78600c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:40.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f7866d64640 1 -- 172.21.15.107:0/3378532475 shutdown_connections 2023-12-04T05:46:40.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f7866d64640 1 -- 172.21.15.107:0/3378532475 wait complete. 2023-12-04T05:46:40.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f7866d64640 1 Processor -- start 2023-12-04T05:46:40.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f7866d64640 1 -- start start 2023-12-04T05:46:40.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f7866d64640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7860072110 0x7f7860136b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:40.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f7866d64640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78600724f0 0x7f786013afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:40.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f7866d64640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f78600c56f0 0x7f786013b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:40.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f7866d64640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7860071c70 con 0x7f78600724f0 2023-12-04T05:46:40.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f7866d64640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7860071430 con 0x7f7860072110 2023-12-04T05:46:40.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f7866d64640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7860136160 con 0x7f78600c56f0 2023-12-04T05:46:40.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f785ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78600724f0 0x7f786013afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:40.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f785ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78600724f0 0x7f786013afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44464/0 (socket says 172.21.15.107:44464) 2023-12-04T05:46:40.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f785ffff640 1 -- 172.21.15.107:0/1734194840 learned_addr learned my addr 172.21.15.107:0/1734194840 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:40.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f7864b00640 1 --2- 172.21.15.107:0/1734194840 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7860072110 0x7f7860136b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:40.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f7865301640 1 --2- 172.21.15.107:0/1734194840 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f78600c56f0 0x7f786013b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:40.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f785ffff640 1 -- 172.21.15.107:0/1734194840 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f78600c56f0 msgr2=0x7f786013b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:40.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f785ffff640 1 --2- 172.21.15.107:0/1734194840 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f78600c56f0 0x7f786013b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:40.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f785ffff640 1 -- 172.21.15.107:0/1734194840 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7860072110 msgr2=0x7f7860136b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:40.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f785ffff640 1 --2- 172.21.15.107:0/1734194840 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7860072110 0x7f7860136b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:40.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f785ffff640 1 -- 172.21.15.107:0/1734194840 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78500094d0 con 0x7f78600724f0 2023-12-04T05:46:40.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f7864b00640 1 --2- 172.21.15.107:0/1734194840 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7860072110 0x7f7860136b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:46:40.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f785ffff640 1 --2- 172.21.15.107:0/1734194840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78600724f0 0x7f786013afc0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f785400c440 tx=0x7f785400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:40.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f785d7fa640 1 -- 172.21.15.107:0/1734194840 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7854012830 con 0x7f78600724f0 2023-12-04T05:46:40.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f785d7fa640 1 -- 172.21.15.107:0/1734194840 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7854012990 con 0x7f78600724f0 2023-12-04T05:46:40.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f7866d64640 1 -- 172.21.15.107:0/1734194840 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78600716b0 con 0x7f78600724f0 2023-12-04T05:46:40.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f785d7fa640 1 -- 172.21.15.107:0/1734194840 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7854012d40 con 0x7f78600724f0 2023-12-04T05:46:40.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.606+0000 7f7866d64640 1 -- 172.21.15.107:0/1734194840 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7860070de0 con 0x7f78600724f0 2023-12-04T05:46:40.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.610+0000 7f7866d64640 1 -- 172.21.15.107:0/1734194840 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7860070c40 con 0x7f78600724f0 2023-12-04T05:46:40.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.610+0000 7f785d7fa640 1 -- 172.21.15.107:0/1734194840 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f785401b400 con 0x7f78600724f0 2023-12-04T05:46:40.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.610+0000 7f785d7fa640 1 --2- 172.21.15.107:0/1734194840 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f784c066900 0x7f784c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:40.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.610+0000 7f785d7fa640 1 -- 172.21.15.107:0/1734194840 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f78540844e0 con 0x7f78600724f0 2023-12-04T05:46:40.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.610+0000 7f7864b00640 1 --2- 172.21.15.107:0/1734194840 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f784c066900 0x7f784c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:40.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.610+0000 7f7864b00640 1 --2- 172.21.15.107:0/1734194840 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f784c066900 0x7f784c068dc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f78500af000 tx=0x7f78500a7070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:40.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.614+0000 7f785d7fa640 1 -- 172.21.15.107:0/1734194840 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7860070c40 con 0x7f78600724f0 2023-12-04T05:46:40.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.798+0000 7f7866d64640 1 -- 172.21.15.107:0/1734194840 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7860070c40 con 0x7f78600724f0 2023-12-04T05:46:40.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.802+0000 7f785d7fa640 1 -- 172.21.15.107:0/1734194840 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f7860070c40 con 0x7f78600724f0 2023-12-04T05:46:40.803 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:40.804 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:40.804 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:46:40.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.802+0000 7f7866d64640 1 -- 172.21.15.107:0/1734194840 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f784c066900 msgr2=0x7f784c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:40.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.802+0000 7f7866d64640 1 --2- 172.21.15.107:0/1734194840 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f784c066900 0x7f784c068dc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f78500af000 tx=0x7f78500a7070 comp rx=0 tx=0).stop 2023-12-04T05:46:40.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.802+0000 7f7866d64640 1 -- 172.21.15.107:0/1734194840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78600724f0 msgr2=0x7f786013afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:40.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.802+0000 7f7866d64640 1 --2- 172.21.15.107:0/1734194840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78600724f0 0x7f786013afc0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f785400c440 tx=0x7f785400c900 comp rx=0 tx=0).stop 2023-12-04T05:46:40.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.802+0000 7f7866d64640 1 -- 172.21.15.107:0/1734194840 shutdown_connections 2023-12-04T05:46:40.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.802+0000 7f7866d64640 1 --2- 172.21.15.107:0/1734194840 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f784c066900 0x7f784c068dc0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:40.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.802+0000 7f7866d64640 1 --2- 172.21.15.107:0/1734194840 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f78600c56f0 0x7f786013b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:40.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.802+0000 7f7866d64640 1 --2- 172.21.15.107:0/1734194840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78600724f0 0x7f786013afc0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:40.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.802+0000 7f7866d64640 1 --2- 172.21.15.107:0/1734194840 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7860072110 0x7f7860136b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:40.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.802+0000 7f7866d64640 1 -- 172.21.15.107:0/1734194840 >> 172.21.15.107:0/1734194840 conn(0x7f78600c7ff0 msgr2=0x7f7860157420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:40.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.802+0000 7f7866d64640 1 -- 172.21.15.107:0/1734194840 shutdown_connections 2023-12-04T05:46:40.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:40.802+0000 7f7866d64640 1 -- 172.21.15.107:0/1734194840 wait complete. 2023-12-04T05:46:40.822 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 50, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-12-04T05:39:59.032019+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_4288": { "addr": "172.21.15.195:6837/3412865477", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6835", "nonce": 3412865477, "type": "v2" }, { "addr": "172.21.15.195:6837", "nonce": 3412865477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_5190": { "addr": "172.21.15.107:6836/2928623624", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 2928623624, "type": "v2" }, { "addr": "172.21.15.107:6836", "nonce": 2928623624, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5190, "incarnation": 45, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:rejoin", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-12-04T05:46:11.040247+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2, 3, 4 ], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 5190 } } } ], "standbys": [ { "addr": "172.21.15.107:6837/2752497349", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6835", "nonce": 2752497349, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 2752497349, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 49, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5292, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-12-04T05:46:41.259 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~60s 2023-12-04T05:46:41.260 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~174s 2023-12-04T05:46:41.260 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~215s 2023-12-04T05:46:42.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:42.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.954+0000 7f1bda3a3640 1 Processor -- start 2023-12-04T05:46:42.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.954+0000 7f1bda3a3640 1 -- start start 2023-12-04T05:46:42.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.954+0000 7f1bda3a3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1bd40d01c0 0x7f1bd4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:42.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.954+0000 7f1bda3a3640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1bd4072220 0x7f1bd40c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:42.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.954+0000 7f1bda3a3640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd415d710 con 0x7f1bd4154020 2023-12-04T05:46:42.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.954+0000 7f1bda3a3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bd40ca230 con 0x7f1bd40d01c0 2023-12-04T05:46:42.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.954+0000 7f1bda3a3640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd40c6770 con 0x7f1bd4072220 2023-12-04T05:46:42.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.954+0000 7f1bd37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1bd40d01c0 0x7f1bd4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:42.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.954+0000 7f1bd37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1bd40d01c0 0x7f1bd4071ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44478/0 (socket says 172.21.15.107:44478) 2023-12-04T05:46:42.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.954+0000 7f1bd37fe640 1 -- 172.21.15.107:0/1635213200 learned_addr learned my addr 172.21.15.107:0/1635213200 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:42.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.954+0000 7f1bd8940640 1 --2- 172.21.15.107:0/1635213200 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1bd4072220 0x7f1bd40c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:42.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.954+0000 7f1bd37fe640 1 -- 172.21.15.107:0/1635213200 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1bd4072220 msgr2=0x7f1bd40c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:42.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.954+0000 7f1bd37fe640 1 --2- 172.21.15.107:0/1635213200 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1bd4072220 0x7f1bd40c4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:42.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.954+0000 7f1bd37fe640 1 -- 172.21.15.107:0/1635213200 >> v1:172.21.15.195:6789/0 conn(0x7f1bd4154020 legacy=0x7f1bd415a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:42.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.954+0000 7f1bd37fe640 1 -- 172.21.15.107:0/1635213200 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd415d9f0 con 0x7f1bd40d01c0 2023-12-04T05:46:42.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bd37fe640 1 --2- 172.21.15.107:0/1635213200 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bd40d01c0 0x7f1bd4071ce0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f1bd406b6d0 tx=0x7f1bc801ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=551aad154049669c server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:42.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bd2ffd640 1 -- 172.21.15.107:0/1635213200 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc80a8970 con 0x7f1bd40d01c0 2023-12-04T05:46:42.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bd2ffd640 1 -- 172.21.15.107:0/1635213200 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc80a8ad0 con 0x7f1bd40d01c0 2023-12-04T05:46:42.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bd2ffd640 1 -- 172.21.15.107:0/1635213200 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc80a8e60 con 0x7f1bd40d01c0 2023-12-04T05:46:42.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bda3a3640 1 -- 172.21.15.107:0/1635213200 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bd40d01c0 msgr2=0x7f1bd4071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:42.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bda3a3640 1 --2- 172.21.15.107:0/1635213200 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bd40d01c0 0x7f1bd4071ce0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f1bd406b6d0 tx=0x7f1bc801ecb0 comp rx=0 tx=0).stop 2023-12-04T05:46:42.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bda3a3640 1 -- 172.21.15.107:0/1635213200 shutdown_connections 2023-12-04T05:46:42.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bda3a3640 1 --2- 172.21.15.107:0/1635213200 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1bd4072220 0x7f1bd40c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:42.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bda3a3640 1 --2- 172.21.15.107:0/1635213200 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bd40d01c0 0x7f1bd4071ce0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:42.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bda3a3640 1 -- 172.21.15.107:0/1635213200 >> 172.21.15.107:0/1635213200 conn(0x7f1bd406c6c0 msgr2=0x7f1bd40cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:42.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bda3a3640 1 -- 172.21.15.107:0/1635213200 shutdown_connections 2023-12-04T05:46:42.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bda3a3640 1 -- 172.21.15.107:0/1635213200 wait complete. 2023-12-04T05:46:42.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bda3a3640 1 Processor -- start 2023-12-04T05:46:42.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bda3a3640 1 -- start start 2023-12-04T05:46:42.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bda3a3640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1bd4072220 0x7f1bd4149b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:42.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bda3a3640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1bd40d01c0 0x7f1bd40cd620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:42.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bda3a3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bd4154020 0x7f1bd40cdb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:42.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bda3a3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd40b4980 con 0x7f1bd4154020 2023-12-04T05:46:42.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bda3a3640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd40cf810 con 0x7f1bd4072220 2023-12-04T05:46:42.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bda3a3640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd40cf570 con 0x7f1bd40d01c0 2023-12-04T05:46:42.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bd8940640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bd4154020 0x7f1bd40cdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:42.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bd8940640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bd4154020 0x7f1bd40cdb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44492/0 (socket says 172.21.15.107:44492) 2023-12-04T05:46:42.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bd8940640 1 -- 172.21.15.107:0/188905520 learned_addr learned my addr 172.21.15.107:0/188905520 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:42.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bd37fe640 1 --2- 172.21.15.107:0/188905520 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1bd40d01c0 0x7f1bd40cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:42.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bd8940640 1 -- 172.21.15.107:0/188905520 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1bd40d01c0 msgr2=0x7f1bd40cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:42.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bd8940640 1 --2- 172.21.15.107:0/188905520 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1bd40d01c0 0x7f1bd40cd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:42.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bd3fff640 1 --2- 172.21.15.107:0/188905520 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1bd4072220 0x7f1bd4149b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:42.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bd8940640 1 -- 172.21.15.107:0/188905520 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1bd4072220 msgr2=0x7f1bd4149b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:42.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bd8940640 1 --2- 172.21.15.107:0/188905520 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1bd4072220 0x7f1bd4149b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:42.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bd8940640 1 -- 172.21.15.107:0/188905520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc80094d0 con 0x7f1bd4154020 2023-12-04T05:46:42.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.958+0000 7f1bd3fff640 1 --2- 172.21.15.107:0/188905520 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1bd4072220 0x7f1bd4149b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:46:42.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.962+0000 7f1bd8940640 1 --2- 172.21.15.107:0/188905520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bd4154020 0x7f1bd40cdb60 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc00cd70 tx=0x7f1bcc00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:42.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.962+0000 7f1bd0ff9640 1 -- 172.21.15.107:0/188905520 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc003ba0 con 0x7f1bd4154020 2023-12-04T05:46:42.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.962+0000 7f1bd0ff9640 1 -- 172.21.15.107:0/188905520 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bcc003d00 con 0x7f1bd4154020 2023-12-04T05:46:42.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.962+0000 7f1bda3a3640 1 -- 172.21.15.107:0/188905520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd4150270 con 0x7f1bd4154020 2023-12-04T05:46:42.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.962+0000 7f1bd0ff9640 1 -- 172.21.15.107:0/188905520 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc004240 con 0x7f1bd4154020 2023-12-04T05:46:42.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.962+0000 7f1bda3a3640 1 -- 172.21.15.107:0/188905520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd41f06b0 con 0x7f1bd4154020 2023-12-04T05:46:42.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.962+0000 7f1bda3a3640 1 -- 172.21.15.107:0/188905520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bd40b4af0 con 0x7f1bd4154020 2023-12-04T05:46:42.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.962+0000 7f1bd0ff9640 1 -- 172.21.15.107:0/188905520 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f1bcc0043a0 con 0x7f1bd4154020 2023-12-04T05:46:42.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.962+0000 7f1bd0ff9640 1 --2- 172.21.15.107:0/188905520 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1bc00668d0 0x7f1bc0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:42.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.962+0000 7f1bd0ff9640 1 -- 172.21.15.107:0/188905520 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f1bcc012070 con 0x7f1bd4154020 2023-12-04T05:46:42.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.962+0000 7f1bd3fff640 1 --2- 172.21.15.107:0/188905520 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1bc00668d0 0x7f1bc0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:42.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.962+0000 7f1bd3fff640 1 --2- 172.21.15.107:0/188905520 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1bc00668d0 0x7f1bc0068d90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1bc400a850 tx=0x7f1bc4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:42.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:42.966+0000 7f1bd0ff9640 1 -- 172.21.15.107:0/188905520 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1bcc052060 con 0x7f1bd4154020 2023-12-04T05:46:43.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:43.146+0000 7f1bda3a3640 1 -- 172.21.15.107:0/188905520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1bd4067870 con 0x7f1bd4154020 2023-12-04T05:46:43.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:43.146+0000 7f1bd0ff9640 1 -- 172.21.15.107:0/188905520 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f1bcc055680 con 0x7f1bd4154020 2023-12-04T05:46:43.150 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:43.151 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:43.151 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:46:43.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:43.150+0000 7f1bda3a3640 1 -- 172.21.15.107:0/188905520 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1bc00668d0 msgr2=0x7f1bc0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:43.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:43.150+0000 7f1bda3a3640 1 --2- 172.21.15.107:0/188905520 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1bc00668d0 0x7f1bc0068d90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1bc400a850 tx=0x7f1bc4008040 comp rx=0 tx=0).stop 2023-12-04T05:46:43.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:43.150+0000 7f1bda3a3640 1 -- 172.21.15.107:0/188905520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bd4154020 msgr2=0x7f1bd40cdb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:43.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:43.150+0000 7f1bda3a3640 1 --2- 172.21.15.107:0/188905520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bd4154020 0x7f1bd40cdb60 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc00cd70 tx=0x7f1bcc00a660 comp rx=0 tx=0).stop 2023-12-04T05:46:43.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:43.150+0000 7f1bda3a3640 1 -- 172.21.15.107:0/188905520 shutdown_connections 2023-12-04T05:46:43.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:43.150+0000 7f1bda3a3640 1 --2- 172.21.15.107:0/188905520 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1bc00668d0 0x7f1bc0068d90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:43.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:43.150+0000 7f1bda3a3640 1 --2- 172.21.15.107:0/188905520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bd4154020 0x7f1bd40cdb60 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:43.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:43.150+0000 7f1bda3a3640 1 --2- 172.21.15.107:0/188905520 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1bd40d01c0 0x7f1bd40cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:43.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:43.150+0000 7f1bda3a3640 1 --2- 172.21.15.107:0/188905520 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1bd4072220 0x7f1bd4149b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:43.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:43.150+0000 7f1bda3a3640 1 -- 172.21.15.107:0/188905520 >> 172.21.15.107:0/188905520 conn(0x7f1bd406c6c0 msgr2=0x7f1bd4152e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:43.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:43.150+0000 7f1bda3a3640 1 -- 172.21.15.107:0/188905520 shutdown_connections 2023-12-04T05:46:43.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:43.150+0000 7f1bda3a3640 1 -- 172.21.15.107:0/188905520 wait complete. 2023-12-04T05:46:43.167 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:46:45.169 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:45.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.302+0000 7fb94c392640 1 Processor -- start 2023-12-04T05:46:45.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.302+0000 7fb94c392640 1 -- start start 2023-12-04T05:46:45.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.302+0000 7fb94c392640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb944154020 0x7fb94415a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:45.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.302+0000 7fb94c392640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb9440d01c0 0x7fb944071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:45.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.302+0000 7fb94c392640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94415ad50 con 0x7fb944072220 2023-12-04T05:46:45.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.302+0000 7fb94c392640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9440ca230 con 0x7fb9440d01c0 2023-12-04T05:46:45.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.302+0000 7fb94c392640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9440c6770 con 0x7fb944154020 2023-12-04T05:46:45.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.302+0000 7fb94992d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb9440d01c0 0x7fb944071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:45.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.302+0000 7fb94992d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb9440d01c0 0x7fb944071ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44496/0 (socket says 172.21.15.107:44496) 2023-12-04T05:46:45.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.302+0000 7fb94992d640 1 -- 172.21.15.107:0/3777913837 learned_addr learned my addr 172.21.15.107:0/3777913837 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:45.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.302+0000 7fb94992d640 1 -- 172.21.15.107:0/3777913837 >> v2:172.21.15.195:3300/0 conn(0x7fb944154020 msgr2=0x7fb94415a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:45.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.302+0000 7fb94a12e640 1 --2- 172.21.15.107:0/3777913837 >> v2:172.21.15.195:3300/0 conn(0x7fb944154020 0x7fb94415a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:45.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.302+0000 7fb94992d640 1 --2- 172.21.15.107:0/3777913837 >> v2:172.21.15.195:3300/0 conn(0x7fb944154020 0x7fb94415a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:45.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.302+0000 7fb94992d640 1 -- 172.21.15.107:0/3777913837 >> v1:172.21.15.195:6789/0 conn(0x7fb944072220 legacy=0x7fb9440c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:45.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.302+0000 7fb94992d640 1 -- 172.21.15.107:0/3777913837 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94415d710 con 0x7fb9440d01c0 2023-12-04T05:46:45.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.302+0000 7fb94a12e640 1 --2- 172.21.15.107:0/3777913837 >> v2:172.21.15.195:3300/0 conn(0x7fb944154020 0x7fb94415a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:46:45.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94992d640 1 --2- 172.21.15.107:0/3777913837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9440d01c0 0x7fb944071ce0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fb9440d0150 tx=0x7fb93401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dcb9d894e794676 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:45.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94912c640 1 -- 172.21.15.107:0/3777913837 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9340a89b0 con 0x7fb9440d01c0 2023-12-04T05:46:45.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94912c640 1 -- 172.21.15.107:0/3777913837 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9340a8b10 con 0x7fb9440d01c0 2023-12-04T05:46:45.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94912c640 1 -- 172.21.15.107:0/3777913837 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9340a8ea0 con 0x7fb9440d01c0 2023-12-04T05:46:45.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94c392640 1 -- 172.21.15.107:0/3777913837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9440d01c0 msgr2=0x7fb944071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:45.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94c392640 1 --2- 172.21.15.107:0/3777913837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9440d01c0 0x7fb944071ce0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fb9440d0150 tx=0x7fb93401ecb0 comp rx=0 tx=0).stop 2023-12-04T05:46:45.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94c392640 1 -- 172.21.15.107:0/3777913837 shutdown_connections 2023-12-04T05:46:45.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94c392640 1 --2- 172.21.15.107:0/3777913837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9440d01c0 0x7fb944071ce0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:45.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94c392640 1 --2- 172.21.15.107:0/3777913837 >> v2:172.21.15.195:3300/0 conn(0x7fb944154020 0x7fb94415a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:45.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94c392640 1 -- 172.21.15.107:0/3777913837 >> 172.21.15.107:0/3777913837 conn(0x7fb94406c6c0 msgr2=0x7fb9440cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:45.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94c392640 1 -- 172.21.15.107:0/3777913837 shutdown_connections 2023-12-04T05:46:45.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94c392640 1 -- 172.21.15.107:0/3777913837 wait complete. 2023-12-04T05:46:45.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94c392640 1 Processor -- start 2023-12-04T05:46:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94c392640 1 -- start start 2023-12-04T05:46:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94c392640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb944072220 0x7fb944158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94c392640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb9440d01c0 0x7fb9441f0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94c392640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb944154020 0x7fb9441f1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94c392640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb944159170 con 0x7fb944154020 2023-12-04T05:46:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94c392640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9441e5ef0 con 0x7fb944072220 2023-12-04T05:46:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94c392640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9441579b0 con 0x7fb9440d01c0 2023-12-04T05:46:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94a92f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb944154020 0x7fb9441f1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94a92f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb944154020 0x7fb9441f1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44498/0 (socket says 172.21.15.107:44498) 2023-12-04T05:46:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94a92f640 1 -- 172.21.15.107:0/1117225357 learned_addr learned my addr 172.21.15.107:0/1117225357 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94a12e640 1 --2- 172.21.15.107:0/1117225357 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb944072220 0x7fb944158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94992d640 1 --2- 172.21.15.107:0/1117225357 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb9440d01c0 0x7fb9441f0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94a92f640 1 -- 172.21.15.107:0/1117225357 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb9440d01c0 msgr2=0x7fb9441f0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94a92f640 1 --2- 172.21.15.107:0/1117225357 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb9440d01c0 0x7fb9441f0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:45.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94a92f640 1 -- 172.21.15.107:0/1117225357 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb944072220 msgr2=0x7fb944158c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:45.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94a92f640 1 --2- 172.21.15.107:0/1117225357 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb944072220 0x7fb944158c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:45.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94a92f640 1 -- 172.21.15.107:0/1117225357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9340094d0 con 0x7fb944154020 2023-12-04T05:46:45.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.306+0000 7fb94a12e640 1 --2- 172.21.15.107:0/1117225357 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb944072220 0x7fb944158c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:46:45.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.310+0000 7fb94a92f640 1 --2- 172.21.15.107:0/1117225357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb944154020 0x7fb9441f1050 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00c7b0 tx=0x7fb93c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:45.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.310+0000 7fb93affd640 1 -- 172.21.15.107:0/1117225357 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c004b70 con 0x7fb944154020 2023-12-04T05:46:45.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.310+0000 7fb93affd640 1 -- 172.21.15.107:0/1117225357 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93c004cd0 con 0x7fb944154020 2023-12-04T05:46:45.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.310+0000 7fb94c392640 1 -- 172.21.15.107:0/1117225357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9441f0b50 con 0x7fb944154020 2023-12-04T05:46:45.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.310+0000 7fb93affd640 1 -- 172.21.15.107:0/1117225357 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c00aa90 con 0x7fb944154020 2023-12-04T05:46:45.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.310+0000 7fb94c392640 1 -- 172.21.15.107:0/1117225357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb944159780 con 0x7fb944154020 2023-12-04T05:46:45.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.310+0000 7fb94c392640 1 -- 172.21.15.107:0/1117225357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb944067870 con 0x7fb944154020 2023-12-04T05:46:45.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.310+0000 7fb93affd640 1 -- 172.21.15.107:0/1117225357 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fb93c00abf0 con 0x7fb944154020 2023-12-04T05:46:45.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.310+0000 7fb93affd640 1 --2- 172.21.15.107:0/1117225357 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb928066900 0x7fb928068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:45.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.310+0000 7fb93affd640 1 -- 172.21.15.107:0/1117225357 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fb93c084800 con 0x7fb944154020 2023-12-04T05:46:45.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.310+0000 7fb94a12e640 1 --2- 172.21.15.107:0/1117225357 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb928066900 0x7fb928068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:45.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.310+0000 7fb94a12e640 1 --2- 172.21.15.107:0/1117225357 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb928066900 0x7fb928068dc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb94000a9b0 tx=0x7fb940008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:45.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.314+0000 7fb93affd640 1 -- 172.21.15.107:0/1117225357 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb93c051870 con 0x7fb944154020 2023-12-04T05:46:45.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.502+0000 7fb94c392640 1 -- 172.21.15.107:0/1117225357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9441f1590 con 0x7fb944154020 2023-12-04T05:46:45.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.502+0000 7fb93affd640 1 -- 172.21.15.107:0/1117225357 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7fb93c054e90 con 0x7fb944154020 2023-12-04T05:46:45.505 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:45.505 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:45.505 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:46:45.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.506+0000 7fb94c392640 1 -- 172.21.15.107:0/1117225357 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb928066900 msgr2=0x7fb928068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:45.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.506+0000 7fb94c392640 1 --2- 172.21.15.107:0/1117225357 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb928066900 0x7fb928068dc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb94000a9b0 tx=0x7fb940008040 comp rx=0 tx=0).stop 2023-12-04T05:46:45.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.506+0000 7fb94c392640 1 -- 172.21.15.107:0/1117225357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb944154020 msgr2=0x7fb9441f1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:45.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.506+0000 7fb94c392640 1 --2- 172.21.15.107:0/1117225357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb944154020 0x7fb9441f1050 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00c7b0 tx=0x7fb93c00cc70 comp rx=0 tx=0).stop 2023-12-04T05:46:45.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.506+0000 7fb94c392640 1 -- 172.21.15.107:0/1117225357 shutdown_connections 2023-12-04T05:46:45.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.506+0000 7fb94c392640 1 --2- 172.21.15.107:0/1117225357 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb928066900 0x7fb928068dc0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:45.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.506+0000 7fb94c392640 1 --2- 172.21.15.107:0/1117225357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb944154020 0x7fb9441f1050 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:45.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.506+0000 7fb94c392640 1 --2- 172.21.15.107:0/1117225357 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb9440d01c0 0x7fb9441f0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:45.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.506+0000 7fb94c392640 1 --2- 172.21.15.107:0/1117225357 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb944072220 0x7fb944158c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:45.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.506+0000 7fb94c392640 1 -- 172.21.15.107:0/1117225357 >> 172.21.15.107:0/1117225357 conn(0x7fb94406c6c0 msgr2=0x7fb9440cc1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:45.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.506+0000 7fb94c392640 1 -- 172.21.15.107:0/1117225357 shutdown_connections 2023-12-04T05:46:45.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:45.506+0000 7fb94c392640 1 -- 172.21.15.107:0/1117225357 wait complete. 2023-12-04T05:46:45.524 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:46:47.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:47.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.658+0000 7f065abbd640 1 Processor -- start 2023-12-04T05:46:47.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.658+0000 7f065abbd640 1 -- start start 2023-12-04T05:46:47.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.658+0000 7f065abbd640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f065406fcb0 0x7f06540cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:47.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.658+0000 7f065abbd640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0654152490 con 0x7f06541520b0 2023-12-04T05:46:47.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.658+0000 7f065abbd640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06540c9ab0 con 0x7f06540c8060 2023-12-04T05:46:47.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.658+0000 7f065abbd640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06540c5fa0 con 0x7f065406fcb0 2023-12-04T05:46:47.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.658+0000 7f0658959640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f06541520b0 0x7f06540cd5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57794/0 (socket says 172.21.15.107:57794) 2023-12-04T05:46:47.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.658+0000 7f0658959640 1 -- 172.21.15.107:0/549560922 learned_addr learned my addr 172.21.15.107:0/549560922 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:46:47.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.658+0000 7f0653fff640 1 --2- 172.21.15.107:0/549560922 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f065406fcb0 0x7f06540cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:47.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.658+0000 7f06537fe640 1 -- 172.21.15.107:0/549560922 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948324849 0 0) 0x7f0654152490 con 0x7f06541520b0 2023-12-04T05:46:47.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f06537fe640 1 -- 172.21.15.107:0/549560922 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f063c003530 con 0x7f06541520b0 2023-12-04T05:46:47.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f06537fe640 1 -- 172.21.15.107:0/549560922 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2240927299 0 0) 0x7f063c003530 con 0x7f06541520b0 2023-12-04T05:46:47.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f06537fe640 1 -- 172.21.15.107:0/549560922 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f065406fcb0 msgr2=0x7f06540cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:47.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f06537fe640 1 --2- 172.21.15.107:0/549560922 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f065406fcb0 0x7f06540cabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:47.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f06537fe640 1 -- 172.21.15.107:0/549560922 >> v1:172.21.15.195:6789/0 conn(0x7f06540c8060 legacy=0x7f0654162570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:47.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f06537fe640 1 -- 172.21.15.107:0/549560922 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06540c9d90 con 0x7f06541520b0 2023-12-04T05:46:47.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f06537fe640 1 -- 172.21.15.107:0/549560922 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f0644001680 con 0x7f06541520b0 2023-12-04T05:46:47.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f06537fe640 1 -- 172.21.15.107:0/549560922 >> v1:172.21.15.107:6789/0 conn(0x7f06541520b0 legacy=0x7f06540cd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:47.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f06537fe640 1 --2- 172.21.15.107:0/549560922 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f063c0039b0 0x7f063c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:47.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f06537fe640 1 --2- 172.21.15.107:0/549560922 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f065406fcb0 0x7f063c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:47.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f06537fe640 1 --2- 172.21.15.107:0/549560922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f063c004d10 0x7f063c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:47.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f06537fe640 1 -- 172.21.15.107:0/549560922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06540c5fa0 con 0x7f063c004d10 2023-12-04T05:46:47.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f06537fe640 1 -- 172.21.15.107:0/549560922 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063c00ba40 con 0x7f063c0039b0 2023-12-04T05:46:47.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f06537fe640 1 -- 172.21.15.107:0/549560922 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f063c00bc00 con 0x7f065406fcb0 2023-12-04T05:46:47.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f065915a640 1 --2- 172.21.15.107:0/549560922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f063c004d10 0x7f063c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:47.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f0658959640 1 --2- 172.21.15.107:0/549560922 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f063c0039b0 0x7f063c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:47.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f0653fff640 1 --2- 172.21.15.107:0/549560922 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f065406fcb0 0x7f063c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:47.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f065915a640 1 -- 172.21.15.107:0/549560922 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f065406fcb0 msgr2=0x7f063c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:47.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f065915a640 1 --2- 172.21.15.107:0/549560922 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f065406fcb0 0x7f063c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:47.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f065915a640 1 -- 172.21.15.107:0/549560922 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f063c0039b0 msgr2=0x7f063c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:47.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f065915a640 1 --2- 172.21.15.107:0/549560922 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f063c0039b0 0x7f063c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:47.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f065915a640 1 -- 172.21.15.107:0/549560922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063c003530 con 0x7f063c004d10 2023-12-04T05:46:47.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f0653fff640 1 --2- 172.21.15.107:0/549560922 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f065406fcb0 0x7f063c0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:46:47.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f065915a640 1 --2- 172.21.15.107:0/549560922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f063c004d10 0x7f063c005190 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f064c001e40 tx=0x7f064c01c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3152c99b9f198f57 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:47.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f06537fe640 1 -- 172.21.15.107:0/549560922 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064c003b80 con 0x7f063c004d10 2023-12-04T05:46:47.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f06537fe640 1 -- 172.21.15.107:0/549560922 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f064c003ce0 con 0x7f063c004d10 2023-12-04T05:46:47.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f06537fe640 1 -- 172.21.15.107:0/549560922 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064c0052c0 con 0x7f063c004d10 2023-12-04T05:46:47.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f065abbd640 1 -- 172.21.15.107:0/549560922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f063c004d10 msgr2=0x7f063c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:47.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f065abbd640 1 --2- 172.21.15.107:0/549560922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f063c004d10 0x7f063c005190 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f064c001e40 tx=0x7f064c01c740 comp rx=0 tx=0).stop 2023-12-04T05:46:47.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f065915a640 1 -- 172.21.15.107:0/549560922 reap_dead start 2023-12-04T05:46:47.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f065abbd640 1 -- 172.21.15.107:0/549560922 shutdown_connections 2023-12-04T05:46:47.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f065abbd640 1 -- 172.21.15.107:0/549560922 >> 172.21.15.107:0/549560922 conn(0x7f065406a5b0 msgr2=0x7f0654071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:47.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f065abbd640 1 -- 172.21.15.107:0/549560922 shutdown_connections 2023-12-04T05:46:47.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.662+0000 7f065abbd640 1 -- 172.21.15.107:0/549560922 wait complete. 2023-12-04T05:46:47.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f065abbd640 1 Processor -- start 2023-12-04T05:46:47.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f065abbd640 1 -- start start 2023-12-04T05:46:47.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f065abbd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0654150cc0 0x7f0654140a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:47.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f065abbd640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0654140110 0x7f0654140f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:47.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f065abbd640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06541f1140 0x7f065413e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:47.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f065abbd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0654141480 con 0x7f0654150cc0 2023-12-04T05:46:47.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f065abbd640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0654140740 con 0x7f06541f1140 2023-12-04T05:46:47.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f065abbd640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06541583c0 con 0x7f0654140110 2023-12-04T05:46:47.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f0658959640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0654150cc0 0x7f0654140a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:47.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f0658959640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0654150cc0 0x7f0654140a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40906/0 (socket says 172.21.15.107:40906) 2023-12-04T05:46:47.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f0658959640 1 -- 172.21.15.107:0/2099286187 learned_addr learned my addr 172.21.15.107:0/2099286187 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:47.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f0653fff640 1 --2- 172.21.15.107:0/2099286187 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0654140110 0x7f0654140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:47.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f065915a640 1 --2- 172.21.15.107:0/2099286187 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06541f1140 0x7f065413e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:47.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f0658959640 1 -- 172.21.15.107:0/2099286187 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0654140110 msgr2=0x7f0654140f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:47.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f0658959640 1 --2- 172.21.15.107:0/2099286187 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0654140110 0x7f0654140f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:47.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f0658959640 1 -- 172.21.15.107:0/2099286187 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06541f1140 msgr2=0x7f065413e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:47.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f0658959640 1 --2- 172.21.15.107:0/2099286187 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06541f1140 0x7f065413e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:47.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f0658959640 1 -- 172.21.15.107:0/2099286187 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06541f1610 con 0x7f0654150cc0 2023-12-04T05:46:47.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f0658959640 1 --2- 172.21.15.107:0/2099286187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0654150cc0 0x7f0654140a00 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f064400eb20 tx=0x7f06440016d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:47.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f06517fa640 1 -- 172.21.15.107:0/2099286187 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064400c870 con 0x7f0654150cc0 2023-12-04T05:46:47.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f06517fa640 1 -- 172.21.15.107:0/2099286187 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f064400c9f0 con 0x7f0654150cc0 2023-12-04T05:46:47.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f065abbd640 1 -- 172.21.15.107:0/2099286187 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f065413ee50 con 0x7f0654150cc0 2023-12-04T05:46:47.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f06517fa640 1 -- 172.21.15.107:0/2099286187 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064400cda0 con 0x7f0654150cc0 2023-12-04T05:46:47.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f065abbd640 1 -- 172.21.15.107:0/2099286187 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0654158660 con 0x7f0654150cc0 2023-12-04T05:46:47.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.666+0000 7f065abbd640 1 -- 172.21.15.107:0/2099286187 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0654067ec0 con 0x7f0654150cc0 2023-12-04T05:46:47.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.670+0000 7f06517fa640 1 -- 172.21.15.107:0/2099286187 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f064401d460 con 0x7f0654150cc0 2023-12-04T05:46:47.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.670+0000 7f06517fa640 1 --2- 172.21.15.107:0/2099286187 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f06400668d0 0x7f0640068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:47.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.670+0000 7f06517fa640 1 -- 172.21.15.107:0/2099286187 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f0644084660 con 0x7f0654150cc0 2023-12-04T05:46:47.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.670+0000 7f0653fff640 1 --2- 172.21.15.107:0/2099286187 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f06400668d0 0x7f0640068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:47.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.670+0000 7f0653fff640 1 --2- 172.21.15.107:0/2099286187 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f06400668d0 0x7f0640068d90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0654144320 tx=0x7f0648008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:47.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.674+0000 7f06517fa640 1 -- 172.21.15.107:0/2099286187 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0644055220 con 0x7f0654150cc0 2023-12-04T05:46:47.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.862+0000 7f065abbd640 1 -- 172.21.15.107:0/2099286187 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06540cb030 con 0x7f0654150cc0 2023-12-04T05:46:47.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.862+0000 7f06517fa640 1 -- 172.21.15.107:0/2099286187 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f0644001530 con 0x7f0654150cc0 2023-12-04T05:46:47.866 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:47.867 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:47.867 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:46:47.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.866+0000 7f065abbd640 1 -- 172.21.15.107:0/2099286187 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f06400668d0 msgr2=0x7f0640068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:47.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.866+0000 7f065abbd640 1 --2- 172.21.15.107:0/2099286187 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f06400668d0 0x7f0640068d90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0654144320 tx=0x7f0648008040 comp rx=0 tx=0).stop 2023-12-04T05:46:47.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.866+0000 7f065abbd640 1 -- 172.21.15.107:0/2099286187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0654150cc0 msgr2=0x7f0654140a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:47.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.866+0000 7f065abbd640 1 --2- 172.21.15.107:0/2099286187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0654150cc0 0x7f0654140a00 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f064400eb20 tx=0x7f06440016d0 comp rx=0 tx=0).stop 2023-12-04T05:46:47.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.866+0000 7f065abbd640 1 -- 172.21.15.107:0/2099286187 shutdown_connections 2023-12-04T05:46:47.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.866+0000 7f065abbd640 1 --2- 172.21.15.107:0/2099286187 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f06400668d0 0x7f0640068d90 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:47.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.866+0000 7f065abbd640 1 --2- 172.21.15.107:0/2099286187 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06541f1140 0x7f065413e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:47.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.866+0000 7f065abbd640 1 --2- 172.21.15.107:0/2099286187 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0654140110 0x7f0654140f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:47.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.866+0000 7f065abbd640 1 --2- 172.21.15.107:0/2099286187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0654150cc0 0x7f0654140a00 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:47.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.866+0000 7f065abbd640 1 -- 172.21.15.107:0/2099286187 >> 172.21.15.107:0/2099286187 conn(0x7f065406a5b0 msgr2=0x7f06540c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:47.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.866+0000 7f065abbd640 1 -- 172.21.15.107:0/2099286187 shutdown_connections 2023-12-04T05:46:47.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:47.866+0000 7f065abbd640 1 -- 172.21.15.107:0/2099286187 wait complete. 2023-12-04T05:46:47.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~66s 2023-12-04T05:46:47.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~180s 2023-12-04T05:46:47.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~221s 2023-12-04T05:46:47.885 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:46:49.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:50.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.010+0000 7f82df991640 1 Processor -- start 2023-12-04T05:46:50.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.010+0000 7f82df991640 1 -- start start 2023-12-04T05:46:50.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82df991640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82d80cb030 0x7f82d80cb460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:50.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82df991640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d815e310 con 0x7f82d8153e90 2023-12-04T05:46:50.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82df991640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d80cd1d0 con 0x7f82d80ccd90 2023-12-04T05:46:50.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82df991640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82d80c6100 con 0x7f82d80cb030 2023-12-04T05:46:50.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82dd72d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f82d8153e90 0x7f82d815a810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57806/0 (socket says 172.21.15.107:57806) 2023-12-04T05:46:50.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82dd72d640 1 -- 172.21.15.107:0/3872291255 learned_addr learned my addr 172.21.15.107:0/3872291255 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:46:50.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82cffff640 1 -- 172.21.15.107:0/3872291255 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411801015 0 0) 0x7f82d815e310 con 0x7f82d8153e90 2023-12-04T05:46:50.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82cffff640 1 -- 172.21.15.107:0/3872291255 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82c4003530 con 0x7f82d8153e90 2023-12-04T05:46:50.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82ddf2e640 1 --2- 172.21.15.107:0/3872291255 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82d80cb030 0x7f82d80cb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:50.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82cffff640 1 -- 172.21.15.107:0/3872291255 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1749478192 0 0) 0x7f82c4003530 con 0x7f82d8153e90 2023-12-04T05:46:50.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82cffff640 1 -- 172.21.15.107:0/3872291255 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82d80cb030 msgr2=0x7f82d80cb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:50.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82cffff640 1 --2- 172.21.15.107:0/3872291255 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82d80cb030 0x7f82d80cb460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:50.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82cffff640 1 -- 172.21.15.107:0/3872291255 >> v1:172.21.15.195:6789/0 conn(0x7f82d80ccd90 legacy=0x7f82d806be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:50.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82cffff640 1 -- 172.21.15.107:0/3872291255 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d8154270 con 0x7f82d8153e90 2023-12-04T05:46:50.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82cffff640 1 -- 172.21.15.107:0/3872291255 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f82d40016a0 con 0x7f82d8153e90 2023-12-04T05:46:50.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82cffff640 1 -- 172.21.15.107:0/3872291255 >> v1:172.21.15.107:6789/0 conn(0x7f82d8153e90 legacy=0x7f82d815a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:50.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82cffff640 1 --2- 172.21.15.107:0/3872291255 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82d80cb030 0x7f82c40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:50.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82cffff640 1 --2- 172.21.15.107:0/3872291255 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82c4003fe0 0x7f82c40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:50.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82cffff640 1 --2- 172.21.15.107:0/3872291255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82c4004d10 0x7f82c4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:50.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82cffff640 1 -- 172.21.15.107:0/3872291255 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c400bbb0 con 0x7f82c4004d10 2023-12-04T05:46:50.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82cffff640 1 -- 172.21.15.107:0/3872291255 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c400bd20 con 0x7f82c4003fe0 2023-12-04T05:46:50.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82cffff640 1 -- 172.21.15.107:0/3872291255 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c400bee0 con 0x7f82d80cb030 2023-12-04T05:46:50.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82ddf2e640 1 --2- 172.21.15.107:0/3872291255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82c4004d10 0x7f82c4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:50.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82dcf2c640 1 --2- 172.21.15.107:0/3872291255 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82c4003fe0 0x7f82c40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:50.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82dd72d640 1 --2- 172.21.15.107:0/3872291255 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82d80cb030 0x7f82c40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:50.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82ddf2e640 1 -- 172.21.15.107:0/3872291255 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82d80cb030 msgr2=0x7f82c40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:50.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82ddf2e640 1 --2- 172.21.15.107:0/3872291255 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82d80cb030 0x7f82c40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:50.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82ddf2e640 1 -- 172.21.15.107:0/3872291255 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82c4003fe0 msgr2=0x7f82c40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:50.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82ddf2e640 1 --2- 172.21.15.107:0/3872291255 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82c4003fe0 0x7f82c40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:50.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82ddf2e640 1 -- 172.21.15.107:0/3872291255 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c4003530 con 0x7f82c4004d10 2023-12-04T05:46:50.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.014+0000 7f82dcf2c640 1 --2- 172.21.15.107:0/3872291255 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82c4003fe0 0x7f82c40047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:46:50.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82ddf2e640 1 --2- 172.21.15.107:0/3872291255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82c4004d10 0x7f82c4005190 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f82d0007360 tx=0x7f82d001ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3565cc7a8744106 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:50.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82cffff640 1 -- 172.21.15.107:0/3872291255 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d00a5d20 con 0x7f82c4004d10 2023-12-04T05:46:50.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82cffff640 1 -- 172.21.15.107:0/3872291255 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d00a5e80 con 0x7f82c4004d10 2023-12-04T05:46:50.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82cffff640 1 -- 172.21.15.107:0/3872291255 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d00ad660 con 0x7f82c4004d10 2023-12-04T05:46:50.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82df991640 1 -- 172.21.15.107:0/3872291255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82c4004d10 msgr2=0x7f82c4005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:50.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82df991640 1 --2- 172.21.15.107:0/3872291255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82c4004d10 0x7f82c4005190 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f82d0007360 tx=0x7f82d001ce20 comp rx=0 tx=0).stop 2023-12-04T05:46:50.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82ddf2e640 1 -- 172.21.15.107:0/3872291255 reap_dead start 2023-12-04T05:46:50.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82df991640 1 -- 172.21.15.107:0/3872291255 shutdown_connections 2023-12-04T05:46:50.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82df991640 1 -- 172.21.15.107:0/3872291255 >> 172.21.15.107:0/3872291255 conn(0x7f82d806b790 msgr2=0x7f82d80663d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:50.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82df991640 1 -- 172.21.15.107:0/3872291255 shutdown_connections 2023-12-04T05:46:50.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82df991640 1 -- 172.21.15.107:0/3872291255 wait complete. 2023-12-04T05:46:50.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82df991640 1 Processor -- start 2023-12-04T05:46:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82df991640 1 -- start start 2023-12-04T05:46:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82df991640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d80b4060 0x7f82d81e5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82df991640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82d80b4440 0x7f82d81e4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82df991640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82d81445f0 0x7f82d81e49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82df991640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d815d140 con 0x7f82d80b4060 2023-12-04T05:46:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82df991640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d8144c60 con 0x7f82d81445f0 2023-12-04T05:46:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82df991640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82d8153770 con 0x7f82d80b4440 2023-12-04T05:46:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82dd72d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d80b4060 0x7f82d81e5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82dd72d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d80b4060 0x7f82d81e5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40936/0 (socket says 172.21.15.107:40936) 2023-12-04T05:46:50.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82dd72d640 1 -- 172.21.15.107:0/2888071914 learned_addr learned my addr 172.21.15.107:0/2888071914 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:50.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82dcf2c640 1 --2- 172.21.15.107:0/2888071914 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82d80b4440 0x7f82d81e4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:50.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82ddf2e640 1 --2- 172.21.15.107:0/2888071914 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82d81445f0 0x7f82d81e49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:50.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82dd72d640 1 -- 172.21.15.107:0/2888071914 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82d80b4440 msgr2=0x7f82d81e4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:50.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82dd72d640 1 --2- 172.21.15.107:0/2888071914 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82d80b4440 0x7f82d81e4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:50.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82dd72d640 1 -- 172.21.15.107:0/2888071914 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82d81445f0 msgr2=0x7f82d81e49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:50.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82dd72d640 1 --2- 172.21.15.107:0/2888071914 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82d81445f0 0x7f82d81e49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:50.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82dd72d640 1 -- 172.21.15.107:0/2888071914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d815cea0 con 0x7f82d80b4060 2023-12-04T05:46:50.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.018+0000 7f82ddf2e640 1 --2- 172.21.15.107:0/2888071914 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82d81445f0 0x7f82d81e49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:46:50.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.022+0000 7f82dd72d640 1 --2- 172.21.15.107:0/2888071914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d80b4060 0x7f82d81e5810 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f82d400eb20 tx=0x7f82d4001620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:50.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.022+0000 7f82cdffb640 1 -- 172.21.15.107:0/2888071914 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d400c870 con 0x7f82d80b4060 2023-12-04T05:46:50.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.022+0000 7f82cdffb640 1 -- 172.21.15.107:0/2888071914 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d400c9d0 con 0x7f82d80b4060 2023-12-04T05:46:50.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.022+0000 7f82df991640 1 -- 172.21.15.107:0/2888071914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d8145d50 con 0x7f82d80b4060 2023-12-04T05:46:50.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.022+0000 7f82cdffb640 1 -- 172.21.15.107:0/2888071914 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d400cd80 con 0x7f82d80b4060 2023-12-04T05:46:50.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.022+0000 7f82df991640 1 -- 172.21.15.107:0/2888071914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d815c200 con 0x7f82d80b4060 2023-12-04T05:46:50.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.022+0000 7f82df991640 1 -- 172.21.15.107:0/2888071914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d8145760 con 0x7f82d80b4060 2023-12-04T05:46:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.022+0000 7f82cdffb640 1 -- 172.21.15.107:0/2888071914 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f82d401d460 con 0x7f82d80b4060 2023-12-04T05:46:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.022+0000 7f82cdffb640 1 --2- 172.21.15.107:0/2888071914 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f82bc0668d0 0x7f82bc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.022+0000 7f82cdffb640 1 -- 172.21.15.107:0/2888071914 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f82d40845a0 con 0x7f82d80b4060 2023-12-04T05:46:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.022+0000 7f82dcf2c640 1 --2- 172.21.15.107:0/2888071914 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f82bc0668d0 0x7f82bc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.022+0000 7f82dcf2c640 1 --2- 172.21.15.107:0/2888071914 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f82bc0668d0 0x7f82bc068d90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f82d80c4e60 tx=0x7f82c8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.026+0000 7f82cdffb640 1 -- 172.21.15.107:0/2888071914 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f82d8145760 con 0x7f82d80b4060 2023-12-04T05:46:50.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.210+0000 7f82df991640 1 -- 172.21.15.107:0/2888071914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82d8145760 con 0x7f82d80b4060 2023-12-04T05:46:50.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.214+0000 7f82cdffb640 1 -- 172.21.15.107:0/2888071914 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f82d8145760 con 0x7f82d80b4060 2023-12-04T05:46:50.215 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:50.215 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:50.215 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:46:50.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.214+0000 7f82df991640 1 -- 172.21.15.107:0/2888071914 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f82bc0668d0 msgr2=0x7f82bc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:50.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.214+0000 7f82df991640 1 --2- 172.21.15.107:0/2888071914 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f82bc0668d0 0x7f82bc068d90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f82d80c4e60 tx=0x7f82c8009040 comp rx=0 tx=0).stop 2023-12-04T05:46:50.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.214+0000 7f82df991640 1 -- 172.21.15.107:0/2888071914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d80b4060 msgr2=0x7f82d81e5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:50.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.214+0000 7f82df991640 1 --2- 172.21.15.107:0/2888071914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d80b4060 0x7f82d81e5810 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f82d400eb20 tx=0x7f82d4001620 comp rx=0 tx=0).stop 2023-12-04T05:46:50.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.214+0000 7f82df991640 1 -- 172.21.15.107:0/2888071914 shutdown_connections 2023-12-04T05:46:50.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.214+0000 7f82df991640 1 --2- 172.21.15.107:0/2888071914 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f82bc0668d0 0x7f82bc068d90 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:50.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.214+0000 7f82df991640 1 --2- 172.21.15.107:0/2888071914 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f82d81445f0 0x7f82d81e49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:50.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.214+0000 7f82df991640 1 --2- 172.21.15.107:0/2888071914 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f82d80b4440 0x7f82d81e4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:50.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.214+0000 7f82df991640 1 --2- 172.21.15.107:0/2888071914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d80b4060 0x7f82d81e5810 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:50.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.214+0000 7f82df991640 1 -- 172.21.15.107:0/2888071914 >> 172.21.15.107:0/2888071914 conn(0x7f82d806b790 msgr2=0x7f82d80cdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:50.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.214+0000 7f82df991640 1 -- 172.21.15.107:0/2888071914 shutdown_connections 2023-12-04T05:46:50.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:50.214+0000 7f82df991640 1 -- 172.21.15.107:0/2888071914 wait complete. 2023-12-04T05:46:50.232 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:46:52.233 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.358+0000 7f410042c640 1 Processor -- start 2023-12-04T05:46:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.358+0000 7f410042c640 1 -- start start 2023-12-04T05:46:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.358+0000 7f410042c640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f40f80c56f0 0x7f40f8158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.358+0000 7f410042c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f8158da0 con 0x7f40f80724f0 2023-12-04T05:46:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.358+0000 7f410042c640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f81583d0 con 0x7f40f8072110 2023-12-04T05:46:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.358+0000 7f410042c640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40f80c6170 con 0x7f40f80c56f0 2023-12-04T05:46:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fd9c7640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f40f80724f0 0x7f40f8140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57820/0 (socket says 172.21.15.107:57820) 2023-12-04T05:46:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fd9c7640 1 -- 172.21.15.107:0/2788402544 learned_addr learned my addr 172.21.15.107:0/2788402544 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:46:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fe9c9640 1 --2- 172.21.15.107:0/2788402544 >> v2:172.21.15.195:3300/0 conn(0x7f40f80c56f0 0x7f40f8158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fd1c6640 1 -- 172.21.15.107:0/2788402544 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734353304 0 0) 0x7f40e8001120 con 0x7f40f80724f0 2023-12-04T05:46:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fd1c6640 1 -- 172.21.15.107:0/2788402544 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e4003530 con 0x7f40f80724f0 2023-12-04T05:46:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fd1c6640 1 -- 172.21.15.107:0/2788402544 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2658528212 0 0) 0x7f40e4003530 con 0x7f40f80724f0 2023-12-04T05:46:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fd1c6640 1 -- 172.21.15.107:0/2788402544 >> v2:172.21.15.195:3300/0 conn(0x7f40f80c56f0 msgr2=0x7f40f8158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fd1c6640 1 --2- 172.21.15.107:0/2788402544 >> v2:172.21.15.195:3300/0 conn(0x7f40f80c56f0 0x7f40f8158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fd1c6640 1 -- 172.21.15.107:0/2788402544 >> v1:172.21.15.195:6789/0 conn(0x7f40f8072110 legacy=0x7f40f80c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fd1c6640 1 -- 172.21.15.107:0/2788402544 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f8157da0 con 0x7f40f80724f0 2023-12-04T05:46:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fd1c6640 1 -- 172.21.15.107:0/2788402544 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f40e8001810 con 0x7f40f80724f0 2023-12-04T05:46:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fd1c6640 1 -- 172.21.15.107:0/2788402544 >> v1:172.21.15.107:6789/0 conn(0x7f40f80724f0 legacy=0x7f40f8140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fd1c6640 1 --2- 172.21.15.107:0/2788402544 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40e4003980 0x7f40e4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fd1c6640 1 --2- 172.21.15.107:0/2788402544 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f40e40047d0 0x7f40e4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fd1c6640 1 --2- 172.21.15.107:0/2788402544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40e4005190 0x7f40e400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fd1c6640 1 -- 172.21.15.107:0/2788402544 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f80c6170 con 0x7f40e4005190 2023-12-04T05:46:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fd1c6640 1 -- 172.21.15.107:0/2788402544 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e400c040 con 0x7f40e4003980 2023-12-04T05:46:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fd1c6640 1 -- 172.21.15.107:0/2788402544 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e400c200 con 0x7f40e40047d0 2023-12-04T05:46:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fe9c9640 1 --2- 172.21.15.107:0/2788402544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40e4005190 0x7f40e400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fe1c8640 1 --2- 172.21.15.107:0/2788402544 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40e4003980 0x7f40e4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fd9c7640 1 --2- 172.21.15.107:0/2788402544 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f40e40047d0 0x7f40e4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fe9c9640 1 -- 172.21.15.107:0/2788402544 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f40e40047d0 msgr2=0x7f40e4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fe9c9640 1 --2- 172.21.15.107:0/2788402544 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f40e40047d0 0x7f40e4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fe9c9640 1 -- 172.21.15.107:0/2788402544 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40e4003980 msgr2=0x7f40e4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fe9c9640 1 --2- 172.21.15.107:0/2788402544 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40e4003980 0x7f40e4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fe9c9640 1 -- 172.21.15.107:0/2788402544 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e4003530 con 0x7f40e4005190 2023-12-04T05:46:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fe9c9640 1 -- 172.21.15.107:0/2788402544 reap_dead start 2023-12-04T05:46:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fe9c9640 1 --2- 172.21.15.107:0/2788402544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40e4005190 0x7f40e400b990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009ae0 tx=0x7f40f401ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d24f38bd54f6a04 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fd1c6640 1 -- 172.21.15.107:0/2788402544 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f4004020 con 0x7f40e4005190 2023-12-04T05:46:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fd1c6640 1 -- 172.21.15.107:0/2788402544 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f4004180 con 0x7f40e4005190 2023-12-04T05:46:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.362+0000 7f40fd1c6640 1 -- 172.21.15.107:0/2788402544 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f40a5680 con 0x7f40e4005190 2023-12-04T05:46:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f410042c640 1 -- 172.21.15.107:0/2788402544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40e4005190 msgr2=0x7f40e400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f410042c640 1 --2- 172.21.15.107:0/2788402544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40e4005190 0x7f40e400b990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009ae0 tx=0x7f40f401ca90 comp rx=0 tx=0).stop 2023-12-04T05:46:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f410042c640 1 -- 172.21.15.107:0/2788402544 shutdown_connections 2023-12-04T05:46:52.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f410042c640 1 --2- 172.21.15.107:0/2788402544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40e4005190 0x7f40e400b990 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:52.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f410042c640 1 -- 172.21.15.107:0/2788402544 >> 172.21.15.107:0/2788402544 conn(0x7f40f80c7ff0 msgr2=0x7f40f80c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:52.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f410042c640 1 -- 172.21.15.107:0/2788402544 shutdown_connections 2023-12-04T05:46:52.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f410042c640 1 -- 172.21.15.107:0/2788402544 wait complete. 2023-12-04T05:46:52.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f410042c640 1 Processor -- start 2023-12-04T05:46:52.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f410042c640 1 -- start start 2023-12-04T05:46:52.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f410042c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40e4005190 0x7f40f8149d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:52.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f410042c640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40f806e420 0x7f40f814a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f410042c640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f40f8159c90 0x7f40f815a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f410042c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f815b320 con 0x7f40e4005190 2023-12-04T05:46:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f410042c640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f8158f60 con 0x7f40f806e420 2023-12-04T05:46:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f410042c640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f815a660 con 0x7f40f8159c90 2023-12-04T05:46:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f40fe1c8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40e4005190 0x7f40f8149d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f40fe1c8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40e4005190 0x7f40f8149d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40956/0 (socket says 172.21.15.107:40956) 2023-12-04T05:46:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f40fe1c8640 1 -- 172.21.15.107:0/2485410980 learned_addr learned my addr 172.21.15.107:0/2485410980 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f40fd9c7640 1 --2- 172.21.15.107:0/2485410980 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40f806e420 0x7f40f814a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f40fe9c9640 1 --2- 172.21.15.107:0/2485410980 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f40f8159c90 0x7f40f815a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f40fe1c8640 1 -- 172.21.15.107:0/2485410980 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f40f8159c90 msgr2=0x7f40f815a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f40fe1c8640 1 --2- 172.21.15.107:0/2485410980 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f40f8159c90 0x7f40f815a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f40fe1c8640 1 -- 172.21.15.107:0/2485410980 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40f806e420 msgr2=0x7f40f814a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f40fe1c8640 1 --2- 172.21.15.107:0/2485410980 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40f806e420 0x7f40f814a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f40fe1c8640 1 -- 172.21.15.107:0/2485410980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f815b500 con 0x7f40e4005190 2023-12-04T05:46:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f40fd9c7640 1 --2- 172.21.15.107:0/2485410980 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40f806e420 0x7f40f814a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:46:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f40fe1c8640 1 --2- 172.21.15.107:0/2485410980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40e4005190 0x7f40f8149d80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f40f000e7a0 tx=0x7f40f000ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f40eeffd640 1 -- 172.21.15.107:0/2485410980 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f000eeb0 con 0x7f40e4005190 2023-12-04T05:46:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f40eeffd640 1 -- 172.21.15.107:0/2485410980 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f0003b90 con 0x7f40e4005190 2023-12-04T05:46:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f410042c640 1 -- 172.21.15.107:0/2485410980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f815b7e0 con 0x7f40e4005190 2023-12-04T05:46:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f40eeffd640 1 -- 172.21.15.107:0/2485410980 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f00050d0 con 0x7f40e4005190 2023-12-04T05:46:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.366+0000 7f410042c640 1 -- 172.21.15.107:0/2485410980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f8159490 con 0x7f40e4005190 2023-12-04T05:46:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.370+0000 7f410042c640 1 -- 172.21.15.107:0/2485410980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f8067870 con 0x7f40e4005190 2023-12-04T05:46:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.370+0000 7f40eeffd640 1 -- 172.21.15.107:0/2485410980 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f40f0005230 con 0x7f40e4005190 2023-12-04T05:46:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.370+0000 7f40eeffd640 1 --2- 172.21.15.107:0/2485410980 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f40dc066900 0x7f40dc068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.370+0000 7f40fd9c7640 1 --2- 172.21.15.107:0/2485410980 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f40dc066900 0x7f40dc068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.370+0000 7f40eeffd640 1 -- 172.21.15.107:0/2485410980 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f40f00844b0 con 0x7f40e4005190 2023-12-04T05:46:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.370+0000 7f40fd9c7640 1 --2- 172.21.15.107:0/2485410980 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f40dc066900 0x7f40dc068dc0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f40e8001280 tx=0x7f40e8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.374+0000 7f40eeffd640 1 -- 172.21.15.107:0/2485410980 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f40f00515b0 con 0x7f40e4005190 2023-12-04T05:46:52.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.562+0000 7f410042c640 1 -- 172.21.15.107:0/2485410980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40f8155750 con 0x7f40e4005190 2023-12-04T05:46:52.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.562+0000 7f40eeffd640 1 -- 172.21.15.107:0/2485410980 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f40f0054bd0 con 0x7f40e4005190 2023-12-04T05:46:52.565 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:52.565 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:52.566 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:46:52.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.566+0000 7f410042c640 1 -- 172.21.15.107:0/2485410980 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f40dc066900 msgr2=0x7f40dc068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:52.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.566+0000 7f410042c640 1 --2- 172.21.15.107:0/2485410980 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f40dc066900 0x7f40dc068dc0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f40e8001280 tx=0x7f40e8009040 comp rx=0 tx=0).stop 2023-12-04T05:46:52.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.566+0000 7f410042c640 1 -- 172.21.15.107:0/2485410980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40e4005190 msgr2=0x7f40f8149d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:52.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.566+0000 7f410042c640 1 --2- 172.21.15.107:0/2485410980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40e4005190 0x7f40f8149d80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f40f000e7a0 tx=0x7f40f000ec60 comp rx=0 tx=0).stop 2023-12-04T05:46:52.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.566+0000 7f410042c640 1 -- 172.21.15.107:0/2485410980 shutdown_connections 2023-12-04T05:46:52.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.566+0000 7f410042c640 1 --2- 172.21.15.107:0/2485410980 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f40dc066900 0x7f40dc068dc0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:52.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.566+0000 7f410042c640 1 --2- 172.21.15.107:0/2485410980 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f40f8159c90 0x7f40f815a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:52.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.566+0000 7f410042c640 1 --2- 172.21.15.107:0/2485410980 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40f806e420 0x7f40f814a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:52.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.566+0000 7f410042c640 1 --2- 172.21.15.107:0/2485410980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40e4005190 0x7f40f8149d80 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:52.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.566+0000 7f410042c640 1 -- 172.21.15.107:0/2485410980 >> 172.21.15.107:0/2485410980 conn(0x7f40f80c7ff0 msgr2=0x7f40f8151470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:52.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.566+0000 7f410042c640 1 -- 172.21.15.107:0/2485410980 shutdown_connections 2023-12-04T05:46:52.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:52.566+0000 7f410042c640 1 -- 172.21.15.107:0/2485410980 wait complete. 2023-12-04T05:46:52.583 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 50, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-12-04T05:39:59.032019+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_4288": { "addr": "172.21.15.195:6837/3412865477", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6835", "nonce": 3412865477, "type": "v2" }, { "addr": "172.21.15.195:6837", "nonce": 3412865477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_5190": { "addr": "172.21.15.107:6836/2928623624", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 2928623624, "type": "v2" }, { "addr": "172.21.15.107:6836", "nonce": 2928623624, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5190, "incarnation": 45, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:rejoin", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-12-04T05:46:11.040247+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2, 3, 4 ], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 5190 } } } ], "standbys": [ { "addr": "172.21.15.107:6837/2752497349", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6835", "nonce": 2752497349, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 2752497349, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 49, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5292, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-12-04T05:46:54.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~73s 2023-12-04T05:46:54.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~187s 2023-12-04T05:46:54.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~228s 2023-12-04T05:46:54.584 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:54.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.710+0000 7fca22940640 1 Processor -- start 2023-12-04T05:46:54.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.710+0000 7fca22940640 1 -- start start 2023-12-04T05:46:54.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.714+0000 7fca22940640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fca1c072110 0x7fca1c158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:54.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.714+0000 7fca22940640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fca1c0c56f0 0x7fca1c140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:54.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.714+0000 7fca22940640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca1c158da0 con 0x7fca1c0724f0 2023-12-04T05:46:54.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.714+0000 7fca22940640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca1c1583d0 con 0x7fca1c0c56f0 2023-12-04T05:46:54.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.714+0000 7fca22940640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c0c6170 con 0x7fca1c072110 2023-12-04T05:46:54.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.714+0000 7fca20edd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fca1c0c56f0 0x7fca1c140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:54.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.714+0000 7fca13fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fca1c0724f0 0x7fca1c0c7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57824/0 (socket says 172.21.15.107:57824) 2023-12-04T05:46:54.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.714+0000 7fca13fff640 1 -- 172.21.15.107:0/1315554754 learned_addr learned my addr 172.21.15.107:0/1315554754 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:46:54.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.714+0000 7fca1bfff640 1 --2- 172.21.15.107:0/1315554754 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fca1c072110 0x7fca1c158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:54.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.714+0000 7fca20edd640 1 -- 172.21.15.107:0/1315554754 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fca1c072110 msgr2=0x7fca1c158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:54.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.714+0000 7fca20edd640 1 --2- 172.21.15.107:0/1315554754 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fca1c072110 0x7fca1c158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:54.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.714+0000 7fca20edd640 1 -- 172.21.15.107:0/1315554754 >> v1:172.21.15.107:6789/0 conn(0x7fca1c0724f0 legacy=0x7fca1c0c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:54.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.714+0000 7fca20edd640 1 -- 172.21.15.107:0/1315554754 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca1c157da0 con 0x7fca1c0c56f0 2023-12-04T05:46:54.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.714+0000 7fca1b7fe640 1 -- 172.21.15.107:0/1315554754 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835504310 0 0) 0x7fca04001120 con 0x7fca1c0724f0 2023-12-04T05:46:54.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.714+0000 7fca20edd640 1 --2- 172.21.15.107:0/1315554754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca1c0c56f0 0x7fca1c140830 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fca14009d90 tx=0x7fca1401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d473fea5c4d9f866 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:54.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.714+0000 7fca1b7fe640 1 -- 172.21.15.107:0/1315554754 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca140a5900 con 0x7fca1c0c56f0 2023-12-04T05:46:54.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.714+0000 7fca1b7fe640 1 -- 172.21.15.107:0/1315554754 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca140a5a60 con 0x7fca1c0c56f0 2023-12-04T05:46:54.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.714+0000 7fca1b7fe640 1 -- 172.21.15.107:0/1315554754 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca140a5df0 con 0x7fca1c0c56f0 2023-12-04T05:46:54.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca22940640 1 -- 172.21.15.107:0/1315554754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca1c0c56f0 msgr2=0x7fca1c140830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:54.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca22940640 1 --2- 172.21.15.107:0/1315554754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca1c0c56f0 0x7fca1c140830 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fca14009d90 tx=0x7fca1401c950 comp rx=0 tx=0).stop 2023-12-04T05:46:54.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca22940640 1 -- 172.21.15.107:0/1315554754 shutdown_connections 2023-12-04T05:46:54.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca22940640 1 --2- 172.21.15.107:0/1315554754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca1c0c56f0 0x7fca1c140830 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:54.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca22940640 1 --2- 172.21.15.107:0/1315554754 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fca1c072110 0x7fca1c158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:54.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca22940640 1 -- 172.21.15.107:0/1315554754 >> 172.21.15.107:0/1315554754 conn(0x7fca1c0c7ff0 msgr2=0x7fca1c0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:54.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca22940640 1 -- 172.21.15.107:0/1315554754 shutdown_connections 2023-12-04T05:46:54.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca22940640 1 -- 172.21.15.107:0/1315554754 wait complete. 2023-12-04T05:46:54.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca22940640 1 Processor -- start 2023-12-04T05:46:54.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca22940640 1 -- start start 2023-12-04T05:46:54.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca22940640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fca1c072110 0x7fca1c1f6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:54.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca22940640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca1c0724f0 0x7fca1c06d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:54.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca22940640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fca1c0c56f0 0x7fca1c06d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:54.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca22940640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c151b50 con 0x7fca1c0724f0 2023-12-04T05:46:54.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca22940640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c1f3b00 con 0x7fca1c072110 2023-12-04T05:46:54.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca22940640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c1555b0 con 0x7fca1c0c56f0 2023-12-04T05:46:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca13fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca1c0724f0 0x7fca1c06d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca13fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca1c0724f0 0x7fca1c06d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40976/0 (socket says 172.21.15.107:40976) 2023-12-04T05:46:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca13fff640 1 -- 172.21.15.107:0/2684713270 learned_addr learned my addr 172.21.15.107:0/2684713270 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca20edd640 1 --2- 172.21.15.107:0/2684713270 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fca1c0c56f0 0x7fca1c06d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca1bfff640 1 --2- 172.21.15.107:0/2684713270 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fca1c072110 0x7fca1c1f6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca13fff640 1 -- 172.21.15.107:0/2684713270 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fca1c0c56f0 msgr2=0x7fca1c06d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca13fff640 1 --2- 172.21.15.107:0/2684713270 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fca1c0c56f0 0x7fca1c06d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca13fff640 1 -- 172.21.15.107:0/2684713270 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fca1c072110 msgr2=0x7fca1c1f6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca13fff640 1 --2- 172.21.15.107:0/2684713270 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fca1c072110 0x7fca1c1f6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca13fff640 1 -- 172.21.15.107:0/2684713270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca14009a70 con 0x7fca1c0724f0 2023-12-04T05:46:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca13fff640 1 --2- 172.21.15.107:0/2684713270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca1c0724f0 0x7fca1c06d0a0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fca0400c780 tx=0x7fca0400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca197fa640 1 -- 172.21.15.107:0/2684713270 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca040072d0 con 0x7fca1c0724f0 2023-12-04T05:46:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca197fa640 1 -- 172.21.15.107:0/2684713270 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca04007430 con 0x7fca1c0724f0 2023-12-04T05:46:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca22940640 1 -- 172.21.15.107:0/2684713270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca1c1f2f80 con 0x7fca1c0724f0 2023-12-04T05:46:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca197fa640 1 -- 172.21.15.107:0/2684713270 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca040047f0 con 0x7fca1c0724f0 2023-12-04T05:46:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.718+0000 7fca22940640 1 -- 172.21.15.107:0/2684713270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca1c1f3540 con 0x7fca1c0724f0 2023-12-04T05:46:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.722+0000 7fca22940640 1 -- 172.21.15.107:0/2684713270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca1c067870 con 0x7fca1c0724f0 2023-12-04T05:46:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.722+0000 7fca197fa640 1 -- 172.21.15.107:0/2684713270 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fca04004950 con 0x7fca1c0724f0 2023-12-04T05:46:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.722+0000 7fca197fa640 1 --2- 172.21.15.107:0/2684713270 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc9fc0668d0 0x7fc9fc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.722+0000 7fca197fa640 1 -- 172.21.15.107:0/2684713270 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fca04083540 con 0x7fca1c0724f0 2023-12-04T05:46:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.722+0000 7fca1bfff640 1 --2- 172.21.15.107:0/2684713270 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc9fc0668d0 0x7fc9fc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.722+0000 7fca1bfff640 1 --2- 172.21.15.107:0/2684713270 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc9fc0668d0 0x7fc9fc068d90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fca0c009790 tx=0x7fca0c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:54.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.726+0000 7fca197fa640 1 -- 172.21.15.107:0/2684713270 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fca04050640 con 0x7fca1c0724f0 2023-12-04T05:46:54.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.910+0000 7fca22940640 1 -- 172.21.15.107:0/2684713270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca1c158da0 con 0x7fca1c0724f0 2023-12-04T05:46:54.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.914+0000 7fca197fa640 1 -- 172.21.15.107:0/2684713270 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7fca1c158da0 con 0x7fca1c0724f0 2023-12-04T05:46:54.915 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:54.915 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:54.915 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:46:54.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.914+0000 7fca22940640 1 -- 172.21.15.107:0/2684713270 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc9fc0668d0 msgr2=0x7fc9fc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:54.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.914+0000 7fca22940640 1 --2- 172.21.15.107:0/2684713270 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc9fc0668d0 0x7fc9fc068d90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fca0c009790 tx=0x7fca0c008040 comp rx=0 tx=0).stop 2023-12-04T05:46:54.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.914+0000 7fca22940640 1 -- 172.21.15.107:0/2684713270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca1c0724f0 msgr2=0x7fca1c06d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:54.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.914+0000 7fca22940640 1 --2- 172.21.15.107:0/2684713270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca1c0724f0 0x7fca1c06d0a0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fca0400c780 tx=0x7fca0400cc40 comp rx=0 tx=0).stop 2023-12-04T05:46:54.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.914+0000 7fca22940640 1 -- 172.21.15.107:0/2684713270 shutdown_connections 2023-12-04T05:46:54.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.914+0000 7fca22940640 1 --2- 172.21.15.107:0/2684713270 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc9fc0668d0 0x7fc9fc068d90 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:54.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.914+0000 7fca22940640 1 --2- 172.21.15.107:0/2684713270 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fca1c0c56f0 0x7fca1c06d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:54.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.914+0000 7fca22940640 1 --2- 172.21.15.107:0/2684713270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca1c0724f0 0x7fca1c06d0a0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:54.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.914+0000 7fca22940640 1 --2- 172.21.15.107:0/2684713270 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fca1c072110 0x7fca1c1f6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:54.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.914+0000 7fca22940640 1 -- 172.21.15.107:0/2684713270 >> 172.21.15.107:0/2684713270 conn(0x7fca1c0c7ff0 msgr2=0x7fca1c157490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:54.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.914+0000 7fca22940640 1 -- 172.21.15.107:0/2684713270 shutdown_connections 2023-12-04T05:46:54.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:54.914+0000 7fca22940640 1 -- 172.21.15.107:0/2684713270 wait complete. 2023-12-04T05:46:54.933 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:46:56.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:57.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.062+0000 7f98d6510640 1 Processor -- start 2023-12-04T05:46:57.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.062+0000 7f98d6510640 1 -- start start 2023-12-04T05:46:57.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.062+0000 7f98d6510640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f98d0072110 0x7f98d0158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:57.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.062+0000 7f98d6510640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d0158da0 con 0x7f98d00724f0 2023-12-04T05:46:57.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.062+0000 7f98d6510640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d01583d0 con 0x7f98d00c56f0 2023-12-04T05:46:57.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.062+0000 7f98d6510640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98d00c6170 con 0x7f98d0072110 2023-12-04T05:46:57.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98cf7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f98d00724f0 0x7f98d00c7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57840/0 (socket says 172.21.15.107:57840) 2023-12-04T05:46:57.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98cf7fe640 1 -- 172.21.15.107:0/2444878251 learned_addr learned my addr 172.21.15.107:0/2444878251 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:46:57.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98ceffd640 1 -- 172.21.15.107:0/2444878251 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626212279 0 0) 0x7f98c4001120 con 0x7f98d00724f0 2023-12-04T05:46:57.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98ceffd640 1 -- 172.21.15.107:0/2444878251 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98b8003530 con 0x7f98d00724f0 2023-12-04T05:46:57.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98cffff640 1 --2- 172.21.15.107:0/2444878251 >> v2:172.21.15.195:3300/0 conn(0x7f98d0072110 0x7f98d0158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:57.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98ceffd640 1 -- 172.21.15.107:0/2444878251 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 438774467 0 0) 0x7f98b8003530 con 0x7f98d00724f0 2023-12-04T05:46:57.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98ceffd640 1 -- 172.21.15.107:0/2444878251 >> v2:172.21.15.195:3300/0 conn(0x7f98d0072110 msgr2=0x7f98d0158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:57.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98ceffd640 1 --2- 172.21.15.107:0/2444878251 >> v2:172.21.15.195:3300/0 conn(0x7f98d0072110 0x7f98d0158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:57.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98ceffd640 1 -- 172.21.15.107:0/2444878251 >> v1:172.21.15.195:6789/0 conn(0x7f98d00c56f0 legacy=0x7f98d0140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:57.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98ceffd640 1 -- 172.21.15.107:0/2444878251 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d0157da0 con 0x7f98d00724f0 2023-12-04T05:46:57.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98ceffd640 1 -- 172.21.15.107:0/2444878251 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f98c4001810 con 0x7f98d00724f0 2023-12-04T05:46:57.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98ceffd640 1 -- 172.21.15.107:0/2444878251 >> v1:172.21.15.107:6789/0 conn(0x7f98d00724f0 legacy=0x7f98d00c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:57.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98ceffd640 1 --2- 172.21.15.107:0/2444878251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b8003980 0x7f98b8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:57.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98ceffd640 1 --2- 172.21.15.107:0/2444878251 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f98b80047d0 0x7f98b8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:57.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98ceffd640 1 --2- 172.21.15.107:0/2444878251 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f98b8005190 0x7f98b800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:57.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98ceffd640 1 -- 172.21.15.107:0/2444878251 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d00c6170 con 0x7f98b8003980 2023-12-04T05:46:57.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98ceffd640 1 -- 172.21.15.107:0/2444878251 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b800c0c0 con 0x7f98b80047d0 2023-12-04T05:46:57.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98ceffd640 1 -- 172.21.15.107:0/2444878251 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b800c280 con 0x7f98b8005190 2023-12-04T05:46:57.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98cffff640 1 --2- 172.21.15.107:0/2444878251 >> v2:172.21.15.195:3300/0 conn(0x7f98d0072110 0x7f98d0158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:46:57.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98cffff640 1 --2- 172.21.15.107:0/2444878251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b8003980 0x7f98b8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:57.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98cf7fe640 1 --2- 172.21.15.107:0/2444878251 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f98b80047d0 0x7f98b8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:57.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98cffff640 1 -- 172.21.15.107:0/2444878251 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f98b8005190 msgr2=0x7f98b800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:57.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98cffff640 1 --2- 172.21.15.107:0/2444878251 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f98b8005190 0x7f98b800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:57.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98cffff640 1 -- 172.21.15.107:0/2444878251 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f98b80047d0 msgr2=0x7f98b8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:46:57.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98cffff640 1 --2- 172.21.15.107:0/2444878251 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f98b80047d0 0x7f98b8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:57.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98cffff640 1 -- 172.21.15.107:0/2444878251 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b8003530 con 0x7f98b8003980 2023-12-04T05:46:57.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98d4aad640 1 -- 172.21.15.107:0/2444878251 reap_dead start 2023-12-04T05:46:57.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98cffff640 1 --2- 172.21.15.107:0/2444878251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b8003980 0x7f98b8003da0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f98c00098b0 tx=0x7f98c001c590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bf2b726e7e296a7 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:57.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98ceffd640 1 -- 172.21.15.107:0/2444878251 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c0004220 con 0x7f98b8003980 2023-12-04T05:46:57.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98ceffd640 1 -- 172.21.15.107:0/2444878251 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c0004380 con 0x7f98b8003980 2023-12-04T05:46:57.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98ceffd640 1 -- 172.21.15.107:0/2444878251 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c0003840 con 0x7f98b8003980 2023-12-04T05:46:57.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98d6510640 1 -- 172.21.15.107:0/2444878251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b8003980 msgr2=0x7f98b8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:57.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.066+0000 7f98d6510640 1 --2- 172.21.15.107:0/2444878251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b8003980 0x7f98b8003da0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f98c00098b0 tx=0x7f98c001c590 comp rx=0 tx=0).stop 2023-12-04T05:46:57.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98d6510640 1 -- 172.21.15.107:0/2444878251 shutdown_connections 2023-12-04T05:46:57.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98d6510640 1 --2- 172.21.15.107:0/2444878251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b8003980 0x7f98b8003da0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:57.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98d6510640 1 -- 172.21.15.107:0/2444878251 >> 172.21.15.107:0/2444878251 conn(0x7f98d00c7ff0 msgr2=0x7f98d00c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:57.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98d6510640 1 -- 172.21.15.107:0/2444878251 shutdown_connections 2023-12-04T05:46:57.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98d6510640 1 -- 172.21.15.107:0/2444878251 wait complete. 2023-12-04T05:46:57.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98d6510640 1 Processor -- start 2023-12-04T05:46:57.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98d6510640 1 -- start start 2023-12-04T05:46:57.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98d6510640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f98b8003980 0x7f98d01ed220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:57.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98d6510640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98d006d580 0x7f98d01eec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:57.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98d6510640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f98d01e1400 0x7f98d01f0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:57.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98d6510640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d01ecd90 con 0x7f98d006d580 2023-12-04T05:46:57.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98d6510640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d01ec1a0 con 0x7f98b8003980 2023-12-04T05:46:57.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98d6510640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d01f1b10 con 0x7f98d01e1400 2023-12-04T05:46:57.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98cf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98d006d580 0x7f98d01eec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:57.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98cf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98d006d580 0x7f98d01eec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40982/0 (socket says 172.21.15.107:40982) 2023-12-04T05:46:57.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98cf7fe640 1 -- 172.21.15.107:0/516264706 learned_addr learned my addr 172.21.15.107:0/516264706 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:57.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98cffff640 1 --2- 172.21.15.107:0/516264706 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f98b8003980 0x7f98d01ed220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:57.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98d4aad640 1 --2- 172.21.15.107:0/516264706 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f98d01e1400 0x7f98d01f0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:57.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98cf7fe640 1 -- 172.21.15.107:0/516264706 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f98d01e1400 msgr2=0x7f98d01f0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:57.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98cf7fe640 1 --2- 172.21.15.107:0/516264706 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f98d01e1400 0x7f98d01f0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:57.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98cf7fe640 1 -- 172.21.15.107:0/516264706 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f98b8003980 msgr2=0x7f98d01ed220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:57.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98cf7fe640 1 --2- 172.21.15.107:0/516264706 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f98b8003980 0x7f98d01ed220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:57.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98cf7fe640 1 -- 172.21.15.107:0/516264706 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d01ec460 con 0x7f98d006d580 2023-12-04T05:46:57.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98d4aad640 1 --2- 172.21.15.107:0/516264706 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f98d01e1400 0x7f98d01f0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:46:57.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98cf7fe640 1 --2- 172.21.15.107:0/516264706 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98d006d580 0x7f98d01eec20 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f98c40117c0 tx=0x7f98c4011c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:57.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98ccff9640 1 -- 172.21.15.107:0/516264706 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c400fb60 con 0x7f98d006d580 2023-12-04T05:46:57.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98ccff9640 1 -- 172.21.15.107:0/516264706 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c400fcc0 con 0x7f98d006d580 2023-12-04T05:46:57.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98d6510640 1 -- 172.21.15.107:0/516264706 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d01ec740 con 0x7f98d006d580 2023-12-04T05:46:57.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98ccff9640 1 -- 172.21.15.107:0/516264706 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4015630 con 0x7f98d006d580 2023-12-04T05:46:57.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98d6510640 1 -- 172.21.15.107:0/516264706 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d006d0a0 con 0x7f98d006d580 2023-12-04T05:46:57.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.070+0000 7f98d6510640 1 -- 172.21.15.107:0/516264706 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d0158040 con 0x7f98d006d580 2023-12-04T05:46:57.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.074+0000 7f98ccff9640 1 -- 172.21.15.107:0/516264706 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f98c4015790 con 0x7f98d006d580 2023-12-04T05:46:57.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.074+0000 7f98ccff9640 1 --2- 172.21.15.107:0/516264706 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f98bc066900 0x7f98bc068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:57.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.074+0000 7f98cffff640 1 --2- 172.21.15.107:0/516264706 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f98bc066900 0x7f98bc068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:57.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.074+0000 7f98ccff9640 1 -- 172.21.15.107:0/516264706 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f98c4087620 con 0x7f98d006d580 2023-12-04T05:46:57.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.074+0000 7f98cffff640 1 --2- 172.21.15.107:0/516264706 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f98bc066900 0x7f98bc068dc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f98c00ae000 tx=0x7f98c00aa040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:57.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.078+0000 7f98ccff9640 1 -- 172.21.15.107:0/516264706 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f98d0158040 con 0x7f98d006d580 2023-12-04T05:46:57.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.266+0000 7f98d6510640 1 -- 172.21.15.107:0/516264706 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98d0158da0 con 0x7f98d006d580 2023-12-04T05:46:57.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.266+0000 7f98ccff9640 1 -- 172.21.15.107:0/516264706 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f98d0158da0 con 0x7f98d006d580 2023-12-04T05:46:57.269 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:57.269 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:57.269 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:46:57.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.270+0000 7f98d6510640 1 -- 172.21.15.107:0/516264706 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f98bc066900 msgr2=0x7f98bc068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:57.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.270+0000 7f98d6510640 1 --2- 172.21.15.107:0/516264706 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f98bc066900 0x7f98bc068dc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f98c00ae000 tx=0x7f98c00aa040 comp rx=0 tx=0).stop 2023-12-04T05:46:57.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.270+0000 7f98d6510640 1 -- 172.21.15.107:0/516264706 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98d006d580 msgr2=0x7f98d01eec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:57.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.270+0000 7f98d6510640 1 --2- 172.21.15.107:0/516264706 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98d006d580 0x7f98d01eec20 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f98c40117c0 tx=0x7f98c4011c80 comp rx=0 tx=0).stop 2023-12-04T05:46:57.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.270+0000 7f98d6510640 1 -- 172.21.15.107:0/516264706 shutdown_connections 2023-12-04T05:46:57.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.270+0000 7f98d6510640 1 --2- 172.21.15.107:0/516264706 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f98bc066900 0x7f98bc068dc0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:57.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.270+0000 7f98d6510640 1 --2- 172.21.15.107:0/516264706 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f98d01e1400 0x7f98d01f0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:57.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.270+0000 7f98d6510640 1 --2- 172.21.15.107:0/516264706 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98d006d580 0x7f98d01eec20 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:57.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.270+0000 7f98d6510640 1 --2- 172.21.15.107:0/516264706 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f98b8003980 0x7f98d01ed220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:57.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.270+0000 7f98d6510640 1 -- 172.21.15.107:0/516264706 >> 172.21.15.107:0/516264706 conn(0x7f98d00c7ff0 msgr2=0x7f98d00c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:57.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.270+0000 7f98d6510640 1 -- 172.21.15.107:0/516264706 shutdown_connections 2023-12-04T05:46:57.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:57.270+0000 7f98d6510640 1 -- 172.21.15.107:0/516264706 wait complete. 2023-12-04T05:46:57.288 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:46:59.289 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:46:59.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.418+0000 7f817c449640 1 Processor -- start 2023-12-04T05:46:59.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.418+0000 7f817c449640 1 -- start start 2023-12-04T05:46:59.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.418+0000 7f817c449640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8174072110 0x7f8174158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:59.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.418+0000 7f817c449640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8174158da0 con 0x7f81740724f0 2023-12-04T05:46:59.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.418+0000 7f817c449640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81741583d0 con 0x7f81740c56f0 2023-12-04T05:46:59.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.418+0000 7f817c449640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81740c6170 con 0x7f8174072110 2023-12-04T05:46:59.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.418+0000 7f817a1e5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8174072110 0x7f8174158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:59.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.418+0000 7f817a1e5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8174072110 0x7f8174158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54634/0 (socket says 172.21.15.107:54634) 2023-12-04T05:46:59.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.418+0000 7f817a1e5640 1 -- 172.21.15.107:0/1035987959 learned_addr learned my addr 172.21.15.107:0/1035987959 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:59.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.418+0000 7f81791e3640 1 -- 172.21.15.107:0/1035987959 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3758762927 0 0) 0x7f8164001120 con 0x7f81740724f0 2023-12-04T05:46:59.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.418+0000 7f81791e3640 1 -- 172.21.15.107:0/1035987959 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8160003530 con 0x7f81740724f0 2023-12-04T05:46:59.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.418+0000 7f817a1e5640 1 -- 172.21.15.107:0/1035987959 >> v1:172.21.15.195:6789/0 conn(0x7f81740c56f0 legacy=0x7f8174140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:59.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.418+0000 7f817a1e5640 1 -- 172.21.15.107:0/1035987959 >> v1:172.21.15.107:6789/0 conn(0x7f81740724f0 legacy=0x7f81740c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:59.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.418+0000 7f817a1e5640 1 -- 172.21.15.107:0/1035987959 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8174157da0 con 0x7f8174072110 2023-12-04T05:46:59.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.422+0000 7f817a1e5640 1 --2- 172.21.15.107:0/1035987959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8174072110 0x7f8174158860 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8170009870 tx=0x7f817001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f6d8eb9e5e942d server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:59.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.422+0000 7f81791e3640 1 -- 172.21.15.107:0/1035987959 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81700a8a30 con 0x7f8174072110 2023-12-04T05:46:59.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.422+0000 7f81791e3640 1 -- 172.21.15.107:0/1035987959 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81700a8b90 con 0x7f8174072110 2023-12-04T05:46:59.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.422+0000 7f81791e3640 1 -- 172.21.15.107:0/1035987959 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81700b0440 con 0x7f8174072110 2023-12-04T05:46:59.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.422+0000 7f817c449640 1 -- 172.21.15.107:0/1035987959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8174072110 msgr2=0x7f8174158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:59.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.422+0000 7f817c449640 1 --2- 172.21.15.107:0/1035987959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8174072110 0x7f8174158860 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8170009870 tx=0x7f817001ed30 comp rx=0 tx=0).stop 2023-12-04T05:46:59.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.422+0000 7f817c449640 1 -- 172.21.15.107:0/1035987959 shutdown_connections 2023-12-04T05:46:59.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.422+0000 7f817c449640 1 --2- 172.21.15.107:0/1035987959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8174072110 0x7f8174158860 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:59.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.422+0000 7f817c449640 1 -- 172.21.15.107:0/1035987959 >> 172.21.15.107:0/1035987959 conn(0x7f81740c7ff0 msgr2=0x7f81740c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:59.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.422+0000 7f817c449640 1 -- 172.21.15.107:0/1035987959 shutdown_connections 2023-12-04T05:46:59.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.422+0000 7f817c449640 1 -- 172.21.15.107:0/1035987959 wait complete. 2023-12-04T05:46:59.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.422+0000 7f817c449640 1 Processor -- start 2023-12-04T05:46:59.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.422+0000 7f817c449640 1 -- start start 2023-12-04T05:46:59.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.422+0000 7f817c449640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8174072110 0x7f81740c7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:59.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.422+0000 7f817c449640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81740724f0 0x7f81741ef910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:59.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.422+0000 7f817c449640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81740c56f0 0x7f81741efe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:59.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.422+0000 7f817c449640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81741ef4c0 con 0x7f81740724f0 2023-12-04T05:46:59.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.422+0000 7f817c449640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817414ccb0 con 0x7f81740c56f0 2023-12-04T05:46:59.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.422+0000 7f817c449640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81741f2bb0 con 0x7f8174072110 2023-12-04T05:46:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.422+0000 7f81799e4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81740724f0 0x7f81741ef910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.422+0000 7f81799e4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81740724f0 0x7f81741ef910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54636/0 (socket says 172.21.15.107:54636) 2023-12-04T05:46:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.426+0000 7f81799e4640 1 -- 172.21.15.107:0/4234613052 learned_addr learned my addr 172.21.15.107:0/4234613052 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:46:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.426+0000 7f817a9e6640 1 --2- 172.21.15.107:0/4234613052 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81740c56f0 0x7f81741efe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.426+0000 7f817a1e5640 1 --2- 172.21.15.107:0/4234613052 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8174072110 0x7f81740c7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.426+0000 7f81799e4640 1 -- 172.21.15.107:0/4234613052 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8174072110 msgr2=0x7f81740c7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.426+0000 7f81799e4640 1 --2- 172.21.15.107:0/4234613052 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8174072110 0x7f81740c7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.426+0000 7f81799e4640 1 -- 172.21.15.107:0/4234613052 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81740c56f0 msgr2=0x7f81741efe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.426+0000 7f81799e4640 1 --2- 172.21.15.107:0/4234613052 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81740c56f0 0x7f81741efe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.426+0000 7f81799e4640 1 -- 172.21.15.107:0/4234613052 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81700094d0 con 0x7f81740724f0 2023-12-04T05:46:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.426+0000 7f817a9e6640 1 --2- 172.21.15.107:0/4234613052 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81740c56f0 0x7f81741efe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:46:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.426+0000 7f81799e4640 1 --2- 172.21.15.107:0/4234613052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81740724f0 0x7f81741ef910 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f816400c730 tx=0x7f816400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.426+0000 7f816affd640 1 -- 172.21.15.107:0/4234613052 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81640052d0 con 0x7f81740724f0 2023-12-04T05:46:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.426+0000 7f816affd640 1 -- 172.21.15.107:0/4234613052 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8164005430 con 0x7f81740724f0 2023-12-04T05:46:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.426+0000 7f817c449640 1 -- 172.21.15.107:0/4234613052 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81741f2e90 con 0x7f81740724f0 2023-12-04T05:46:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.426+0000 7f816affd640 1 -- 172.21.15.107:0/4234613052 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816400a860 con 0x7f81740724f0 2023-12-04T05:46:59.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.426+0000 7f817c449640 1 -- 172.21.15.107:0/4234613052 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81741f3450 con 0x7f81740724f0 2023-12-04T05:46:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.426+0000 7f817c449640 1 -- 172.21.15.107:0/4234613052 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81741ef630 con 0x7f81740724f0 2023-12-04T05:46:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.426+0000 7f816affd640 1 -- 172.21.15.107:0/4234613052 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f816400a9c0 con 0x7f81740724f0 2023-12-04T05:46:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.426+0000 7f816affd640 1 --2- 172.21.15.107:0/4234613052 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f81580668d0 0x7f8158068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:46:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.426+0000 7f817a1e5640 1 --2- 172.21.15.107:0/4234613052 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f81580668d0 0x7f8158068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:46:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.426+0000 7f816affd640 1 -- 172.21.15.107:0/4234613052 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f81640840b0 con 0x7f81740724f0 2023-12-04T05:46:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.426+0000 7f817a1e5640 1 --2- 172.21.15.107:0/4234613052 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f81580668d0 0x7f8158068d90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f81700b2000 tx=0x7f81700aa070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:46:59.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.430+0000 7f816affd640 1 -- 172.21.15.107:0/4234613052 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f81741ef630 con 0x7f81740724f0 2023-12-04T05:46:59.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.614+0000 7f817c449640 1 -- 172.21.15.107:0/4234613052 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8174158da0 con 0x7f81740724f0 2023-12-04T05:46:59.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.614+0000 7f816affd640 1 -- 172.21.15.107:0/4234613052 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f8174158da0 con 0x7f81740724f0 2023-12-04T05:46:59.618 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:46:59.618 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:46:59.618 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:46:59.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.618+0000 7f817c449640 1 -- 172.21.15.107:0/4234613052 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f81580668d0 msgr2=0x7f8158068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:59.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.618+0000 7f817c449640 1 --2- 172.21.15.107:0/4234613052 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f81580668d0 0x7f8158068d90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f81700b2000 tx=0x7f81700aa070 comp rx=0 tx=0).stop 2023-12-04T05:46:59.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.618+0000 7f817c449640 1 -- 172.21.15.107:0/4234613052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81740724f0 msgr2=0x7f81741ef910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:46:59.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.618+0000 7f817c449640 1 --2- 172.21.15.107:0/4234613052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81740724f0 0x7f81741ef910 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f816400c730 tx=0x7f816400cbf0 comp rx=0 tx=0).stop 2023-12-04T05:46:59.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.618+0000 7f817c449640 1 -- 172.21.15.107:0/4234613052 shutdown_connections 2023-12-04T05:46:59.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.618+0000 7f817c449640 1 --2- 172.21.15.107:0/4234613052 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f81580668d0 0x7f8158068d90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:59.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.618+0000 7f817c449640 1 --2- 172.21.15.107:0/4234613052 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f81740c56f0 0x7f81741efe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:59.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.618+0000 7f817c449640 1 --2- 172.21.15.107:0/4234613052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81740724f0 0x7f81741ef910 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:59.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.618+0000 7f817c449640 1 --2- 172.21.15.107:0/4234613052 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8174072110 0x7f81740c7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:46:59.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.618+0000 7f817c449640 1 -- 172.21.15.107:0/4234613052 >> 172.21.15.107:0/4234613052 conn(0x7f81740c7ff0 msgr2=0x7f81740c6340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:46:59.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.618+0000 7f817c449640 1 -- 172.21.15.107:0/4234613052 shutdown_connections 2023-12-04T05:46:59.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:46:59.618+0000 7f817c449640 1 -- 172.21.15.107:0/4234613052 wait complete. 2023-12-04T05:46:59.635 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:47:01.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~79s 2023-12-04T05:47:01.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~193s 2023-12-04T05:47:01.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~234s 2023-12-04T05:47:01.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:47:01.157 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:47:01.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:01.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.766+0000 7f185c6f0640 1 Processor -- start 2023-12-04T05:47:01.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.766+0000 7f185c6f0640 1 -- start start 2023-12-04T05:47:01.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.770+0000 7f185c6f0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1854154290 0x7f185406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:01.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.770+0000 7f185c6f0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185406e320 con 0x7f18540c8c80 2023-12-04T05:47:01.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.770+0000 7f185c6f0640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18540c66e0 con 0x7f18540c6b70 2023-12-04T05:47:01.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.770+0000 7f185c6f0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1854071c10 con 0x7f1854154290 2023-12-04T05:47:01.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.770+0000 7f185a48c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1854154290 0x7f185406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:01.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.770+0000 7f1859c8b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f18540c8c80 0x7f185415a000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60712/0 (socket says 172.21.15.107:60712) 2023-12-04T05:47:01.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.770+0000 7f1859c8b640 1 -- 172.21.15.107:0/1729422829 learned_addr learned my addr 172.21.15.107:0/1729422829 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:47:01.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.770+0000 7f185a48c640 1 -- 172.21.15.107:0/1729422829 >> v1:172.21.15.195:6789/0 conn(0x7f18540c6b70 legacy=0x7f185415a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:01.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.770+0000 7f185a48c640 1 -- 172.21.15.107:0/1729422829 >> v1:172.21.15.107:6789/0 conn(0x7f18540c8c80 legacy=0x7f185415a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:01.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.770+0000 7f185948a640 1 -- 172.21.15.107:0/1729422829 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2880899467 0 0) 0x7f185406e320 con 0x7f18540c8c80 2023-12-04T05:47:01.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.770+0000 7f185a48c640 1 -- 172.21.15.107:0/1729422829 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18540c6fc0 con 0x7f1854154290 2023-12-04T05:47:01.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.770+0000 7f185a48c640 1 --2- 172.21.15.107:0/1729422829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1854154290 0x7f185406fca0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f18500097c0 tx=0x7f185001c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=848a74de29d5b5ab server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:01.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.770+0000 7f185948a640 1 -- 172.21.15.107:0/1729422829 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18500ab600 con 0x7f1854154290 2023-12-04T05:47:01.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.770+0000 7f185948a640 1 -- 172.21.15.107:0/1729422829 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18500a7050 con 0x7f1854154290 2023-12-04T05:47:01.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.770+0000 7f185948a640 1 -- 172.21.15.107:0/1729422829 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18500ab980 con 0x7f1854154290 2023-12-04T05:47:01.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.770+0000 7f185c6f0640 1 -- 172.21.15.107:0/1729422829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1854154290 msgr2=0x7f185406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:01.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.770+0000 7f185c6f0640 1 --2- 172.21.15.107:0/1729422829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1854154290 0x7f185406fca0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f18500097c0 tx=0x7f185001c320 comp rx=0 tx=0).stop 2023-12-04T05:47:01.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185c6f0640 1 -- 172.21.15.107:0/1729422829 shutdown_connections 2023-12-04T05:47:01.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185c6f0640 1 --2- 172.21.15.107:0/1729422829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1854154290 0x7f185406fca0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:01.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185c6f0640 1 -- 172.21.15.107:0/1729422829 >> 172.21.15.107:0/1729422829 conn(0x7f18540c7ff0 msgr2=0x7f18540c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:01.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185c6f0640 1 -- 172.21.15.107:0/1729422829 shutdown_connections 2023-12-04T05:47:01.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185c6f0640 1 -- 172.21.15.107:0/1729422829 wait complete. 2023-12-04T05:47:01.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185c6f0640 1 Processor -- start 2023-12-04T05:47:01.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185c6f0640 1 -- start start 2023-12-04T05:47:01.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185c6f0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18540c6b70 0x7f18541598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:01.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185c6f0640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18540c8c80 0x7f1854146490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:01.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185c6f0640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1854154290 0x7f1854142850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:01.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185c6f0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1854159e10 con 0x7f18540c6b70 2023-12-04T05:47:01.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185c6f0640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1854146000 con 0x7f18540c8c80 2023-12-04T05:47:01.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185c6f0640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1854145170 con 0x7f1854154290 2023-12-04T05:47:01.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185a48c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18540c6b70 0x7f18541598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:01.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185a48c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18540c6b70 0x7f18541598d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54646/0 (socket says 172.21.15.107:54646) 2023-12-04T05:47:01.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185a48c640 1 -- 172.21.15.107:0/825404506 learned_addr learned my addr 172.21.15.107:0/825404506 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:01.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f1859c8b640 1 --2- 172.21.15.107:0/825404506 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18540c8c80 0x7f1854146490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:01.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185a48c640 1 -- 172.21.15.107:0/825404506 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1854154290 msgr2=0x7f1854142850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:01.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185a48c640 1 --2- 172.21.15.107:0/825404506 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1854154290 0x7f1854142850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:01.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185a48c640 1 -- 172.21.15.107:0/825404506 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18540c8c80 msgr2=0x7f1854146490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:01.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185a48c640 1 --2- 172.21.15.107:0/825404506 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18540c8c80 0x7f1854146490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:01.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185a48c640 1 -- 172.21.15.107:0/825404506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1850009420 con 0x7f18540c6b70 2023-12-04T05:47:01.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185a48c640 1 --2- 172.21.15.107:0/825404506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18540c6b70 0x7f18541598d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f18500ae040 tx=0x7f185001ab60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:01.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f184affd640 1 -- 172.21.15.107:0/825404506 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18500b7650 con 0x7f18540c6b70 2023-12-04T05:47:01.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f184affd640 1 -- 172.21.15.107:0/825404506 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18500b77d0 con 0x7f18540c6b70 2023-12-04T05:47:01.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185c6f0640 1 -- 172.21.15.107:0/825404506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1854145410 con 0x7f18540c6b70 2023-12-04T05:47:01.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f184affd640 1 -- 172.21.15.107:0/825404506 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18500b7b80 con 0x7f18540c6b70 2023-12-04T05:47:01.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185c6f0640 1 -- 172.21.15.107:0/825404506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18541456d0 con 0x7f18540c6b70 2023-12-04T05:47:01.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.774+0000 7f185c6f0640 1 -- 172.21.15.107:0/825404506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1854072ff0 con 0x7f18540c6b70 2023-12-04T05:47:01.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.778+0000 7f184affd640 1 -- 172.21.15.107:0/825404506 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f18500b7ce0 con 0x7f18540c6b70 2023-12-04T05:47:01.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.778+0000 7f184affd640 1 --2- 172.21.15.107:0/825404506 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f18380668d0 0x7f1838068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:01.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.778+0000 7f184affd640 1 -- 172.21.15.107:0/825404506 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f185011f290 con 0x7f18540c6b70 2023-12-04T05:47:01.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.778+0000 7f1859c8b640 1 --2- 172.21.15.107:0/825404506 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f18380668d0 0x7f1838068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:01.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.778+0000 7f1859c8b640 1 --2- 172.21.15.107:0/825404506 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f18380668d0 0x7f1838068d90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1854157bb0 tx=0x7f1844009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:01.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.782+0000 7f184affd640 1 -- 172.21.15.107:0/825404506 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f18500ec470 con 0x7f18540c6b70 2023-12-04T05:47:01.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.962+0000 7f185c6f0640 1 -- 172.21.15.107:0/825404506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1854067870 con 0x7f18540c6b70 2023-12-04T05:47:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.962+0000 7f184affd640 1 -- 172.21.15.107:0/825404506 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f18500efa90 con 0x7f18540c6b70 2023-12-04T05:47:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:01.965 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:01.966 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:01.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.966+0000 7f185c6f0640 1 -- 172.21.15.107:0/825404506 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f18380668d0 msgr2=0x7f1838068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:01.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.966+0000 7f185c6f0640 1 --2- 172.21.15.107:0/825404506 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f18380668d0 0x7f1838068d90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1854157bb0 tx=0x7f1844009040 comp rx=0 tx=0).stop 2023-12-04T05:47:01.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.966+0000 7f185c6f0640 1 -- 172.21.15.107:0/825404506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18540c6b70 msgr2=0x7f18541598d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:01.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.966+0000 7f185c6f0640 1 --2- 172.21.15.107:0/825404506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18540c6b70 0x7f18541598d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f18500ae040 tx=0x7f185001ab60 comp rx=0 tx=0).stop 2023-12-04T05:47:01.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.966+0000 7f185c6f0640 1 -- 172.21.15.107:0/825404506 shutdown_connections 2023-12-04T05:47:01.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.966+0000 7f185c6f0640 1 --2- 172.21.15.107:0/825404506 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f18380668d0 0x7f1838068d90 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:01.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.966+0000 7f185c6f0640 1 --2- 172.21.15.107:0/825404506 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1854154290 0x7f1854142850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:01.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.966+0000 7f185c6f0640 1 --2- 172.21.15.107:0/825404506 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18540c8c80 0x7f1854146490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:01.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.966+0000 7f185c6f0640 1 --2- 172.21.15.107:0/825404506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18540c6b70 0x7f18541598d0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:01.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.966+0000 7f185c6f0640 1 -- 172.21.15.107:0/825404506 >> 172.21.15.107:0/825404506 conn(0x7f18540c7ff0 msgr2=0x7f185406fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:01.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.966+0000 7f185c6f0640 1 -- 172.21.15.107:0/825404506 shutdown_connections 2023-12-04T05:47:01.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:01.966+0000 7f185c6f0640 1 -- 172.21.15.107:0/825404506 wait complete. 2023-12-04T05:47:01.983 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:47:03.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:04.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.114+0000 7fae71d63640 1 Processor -- start 2023-12-04T05:47:04.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.114+0000 7fae71d63640 1 -- start start 2023-12-04T05:47:04.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.114+0000 7fae71d63640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fae6c0c79e0 0x7fae6c070a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:04.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.114+0000 7fae71d63640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae6c0700c0 con 0x7fae6c151180 2023-12-04T05:47:04.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.114+0000 7fae71d63640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae6c0c7de0 con 0x7fae6c071c00 2023-12-04T05:47:04.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.114+0000 7fae71d63640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae6c156df0 con 0x7fae6c0c79e0 2023-12-04T05:47:04.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.114+0000 7fae6bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fae6c0c79e0 0x7fae6c070a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:04.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.114+0000 7fae6b7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fae6c151180 0x7fae6c0714f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60726/0 (socket says 172.21.15.107:60726) 2023-12-04T05:47:04.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.114+0000 7fae6b7fe640 1 -- 172.21.15.107:0/1464882519 learned_addr learned my addr 172.21.15.107:0/1464882519 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:47:04.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.114+0000 7fae6bfff640 1 -- 172.21.15.107:0/1464882519 >> v1:172.21.15.195:6789/0 conn(0x7fae6c071c00 legacy=0x7fae6c070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:04.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.114+0000 7fae6bfff640 1 -- 172.21.15.107:0/1464882519 >> v1:172.21.15.107:6789/0 conn(0x7fae6c151180 legacy=0x7fae6c0714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:04.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.114+0000 7fae6bfff640 1 -- 172.21.15.107:0/1464882519 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae6c1570d0 con 0x7fae6c0c79e0 2023-12-04T05:47:04.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae6bfff640 1 --2- 172.21.15.107:0/1464882519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae6c0c79e0 0x7fae6c070a90 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fae64009d90 tx=0x7fae6401f100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88aab8365d9f7afc server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:04.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae6affd640 1 -- 172.21.15.107:0/1464882519 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae640af600 con 0x7fae6c0c79e0 2023-12-04T05:47:04.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae6affd640 1 -- 172.21.15.107:0/1464882519 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae640ab050 con 0x7fae6c0c79e0 2023-12-04T05:47:04.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae6affd640 1 -- 172.21.15.107:0/1464882519 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae640af980 con 0x7fae6c0c79e0 2023-12-04T05:47:04.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae71d63640 1 -- 172.21.15.107:0/1464882519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae6c0c79e0 msgr2=0x7fae6c070a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:04.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae71d63640 1 --2- 172.21.15.107:0/1464882519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae6c0c79e0 0x7fae6c070a90 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fae64009d90 tx=0x7fae6401f100 comp rx=0 tx=0).stop 2023-12-04T05:47:04.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae71d63640 1 -- 172.21.15.107:0/1464882519 shutdown_connections 2023-12-04T05:47:04.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae71d63640 1 --2- 172.21.15.107:0/1464882519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae6c0c79e0 0x7fae6c070a90 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:04.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae71d63640 1 -- 172.21.15.107:0/1464882519 >> 172.21.15.107:0/1464882519 conn(0x7fae6c0c7ff0 msgr2=0x7fae6c0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:04.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae71d63640 1 -- 172.21.15.107:0/1464882519 shutdown_connections 2023-12-04T05:47:04.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae71d63640 1 -- 172.21.15.107:0/1464882519 wait complete. 2023-12-04T05:47:04.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae71d63640 1 Processor -- start 2023-12-04T05:47:04.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae71d63640 1 -- start start 2023-12-04T05:47:04.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae71d63640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fae6c071c00 0x7fae6c0c3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:04.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae71d63640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae6c0c79e0 0x7fae6c0c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:04.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae71d63640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fae6c151180 0x7fae6c0c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:04.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae71d63640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6c150280 con 0x7fae6c0c79e0 2023-12-04T05:47:04.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae71d63640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6c14ff50 con 0x7fae6c071c00 2023-12-04T05:47:04.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae71d63640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae6c14f390 con 0x7fae6c151180 2023-12-04T05:47:04.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae62ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae6c0c79e0 0x7fae6c0c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:04.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae62ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae6c0c79e0 0x7fae6c0c64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54672/0 (socket says 172.21.15.107:54672) 2023-12-04T05:47:04.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae62ffd640 1 -- 172.21.15.107:0/457918569 learned_addr learned my addr 172.21.15.107:0/457918569 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:04.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae6bfff640 1 --2- 172.21.15.107:0/457918569 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fae6c151180 0x7fae6c0c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:04.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae6b7fe640 1 --2- 172.21.15.107:0/457918569 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fae6c071c00 0x7fae6c0c3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:04.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae62ffd640 1 -- 172.21.15.107:0/457918569 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fae6c151180 msgr2=0x7fae6c0c55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:04.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae62ffd640 1 --2- 172.21.15.107:0/457918569 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fae6c151180 0x7fae6c0c55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:04.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae62ffd640 1 -- 172.21.15.107:0/457918569 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fae6c071c00 msgr2=0x7fae6c0c3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:04.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae62ffd640 1 --2- 172.21.15.107:0/457918569 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fae6c071c00 0x7fae6c0c3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:04.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae62ffd640 1 -- 172.21.15.107:0/457918569 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae64009a70 con 0x7fae6c0c79e0 2023-12-04T05:47:04.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae62ffd640 1 --2- 172.21.15.107:0/457918569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae6c0c79e0 0x7fae6c0c64f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fae5400b410 tx=0x7fae5400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:04.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.118+0000 7fae68ff9640 1 -- 172.21.15.107:0/457918569 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae54004020 con 0x7fae6c0c79e0 2023-12-04T05:47:04.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.122+0000 7fae68ff9640 1 -- 172.21.15.107:0/457918569 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae54004180 con 0x7fae6c0c79e0 2023-12-04T05:47:04.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.122+0000 7fae71d63640 1 -- 172.21.15.107:0/457918569 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae6c14f670 con 0x7fae6c0c79e0 2023-12-04T05:47:04.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.122+0000 7fae68ff9640 1 -- 172.21.15.107:0/457918569 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae54011630 con 0x7fae6c0c79e0 2023-12-04T05:47:04.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.122+0000 7fae71d63640 1 -- 172.21.15.107:0/457918569 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae6c14fc30 con 0x7fae6c0c79e0 2023-12-04T05:47:04.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.122+0000 7fae71d63640 1 -- 172.21.15.107:0/457918569 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae6c1503f0 con 0x7fae6c0c79e0 2023-12-04T05:47:04.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.122+0000 7fae68ff9640 1 -- 172.21.15.107:0/457918569 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fae54011790 con 0x7fae6c0c79e0 2023-12-04T05:47:04.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.122+0000 7fae68ff9640 1 --2- 172.21.15.107:0/457918569 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fae4c0668d0 0x7fae4c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:04.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.122+0000 7fae6b7fe640 1 --2- 172.21.15.107:0/457918569 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fae4c0668d0 0x7fae4c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:04.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.122+0000 7fae68ff9640 1 -- 172.21.15.107:0/457918569 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fae54084400 con 0x7fae6c0c79e0 2023-12-04T05:47:04.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.122+0000 7fae6b7fe640 1 --2- 172.21.15.107:0/457918569 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fae4c0668d0 0x7fae4c068d90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fae5c005890 tx=0x7fae5c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:04.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.126+0000 7fae68ff9640 1 -- 172.21.15.107:0/457918569 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fae540513e0 con 0x7fae6c0c79e0 2023-12-04T05:47:04.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.318+0000 7fae71d63640 1 -- 172.21.15.107:0/457918569 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae6c067870 con 0x7fae6c0c79e0 2023-12-04T05:47:04.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.318+0000 7fae68ff9640 1 -- 172.21.15.107:0/457918569 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7fae54054a00 con 0x7fae6c0c79e0 2023-12-04T05:47:04.320 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:04.320 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:04.320 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:04.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.318+0000 7fae71d63640 1 -- 172.21.15.107:0/457918569 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fae4c0668d0 msgr2=0x7fae4c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:04.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.318+0000 7fae71d63640 1 --2- 172.21.15.107:0/457918569 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fae4c0668d0 0x7fae4c068d90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fae5c005890 tx=0x7fae5c009040 comp rx=0 tx=0).stop 2023-12-04T05:47:04.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.318+0000 7fae71d63640 1 -- 172.21.15.107:0/457918569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae6c0c79e0 msgr2=0x7fae6c0c64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:04.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.318+0000 7fae71d63640 1 --2- 172.21.15.107:0/457918569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae6c0c79e0 0x7fae6c0c64f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fae5400b410 tx=0x7fae5400b8d0 comp rx=0 tx=0).stop 2023-12-04T05:47:04.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.318+0000 7fae71d63640 1 -- 172.21.15.107:0/457918569 shutdown_connections 2023-12-04T05:47:04.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.318+0000 7fae71d63640 1 --2- 172.21.15.107:0/457918569 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fae4c0668d0 0x7fae4c068d90 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:04.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.318+0000 7fae71d63640 1 --2- 172.21.15.107:0/457918569 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fae6c151180 0x7fae6c0c55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:04.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.318+0000 7fae71d63640 1 --2- 172.21.15.107:0/457918569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae6c0c79e0 0x7fae6c0c64f0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:04.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.318+0000 7fae71d63640 1 --2- 172.21.15.107:0/457918569 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fae6c071c00 0x7fae6c0c3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:04.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.318+0000 7fae71d63640 1 -- 172.21.15.107:0/457918569 >> 172.21.15.107:0/457918569 conn(0x7fae6c0c7ff0 msgr2=0x7fae6c06d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:04.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.318+0000 7fae71d63640 1 -- 172.21.15.107:0/457918569 shutdown_connections 2023-12-04T05:47:04.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:04.318+0000 7fae71d63640 1 -- 172.21.15.107:0/457918569 wait complete. 2023-12-04T05:47:04.338 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 50, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-12-04T05:39:59.032019+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_4288": { "addr": "172.21.15.195:6837/3412865477", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6835", "nonce": 3412865477, "type": "v2" }, { "addr": "172.21.15.195:6837", "nonce": 3412865477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_5190": { "addr": "172.21.15.107:6836/2928623624", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 2928623624, "type": "v2" }, { "addr": "172.21.15.107:6836", "nonce": 2928623624, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5190, "incarnation": 45, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:rejoin", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-12-04T05:46:11.040247+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2, 3, 4 ], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 5190 } } } ], "standbys": [ { "addr": "172.21.15.107:6837/2752497349", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6835", "nonce": 2752497349, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 2752497349, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 49, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5292, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-12-04T05:47:06.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:06.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.466+0000 7f59e6659640 1 Processor -- start 2023-12-04T05:47:06.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.466+0000 7f59e6659640 1 -- start start 2023-12-04T05:47:06.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.466+0000 7f59e6659640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f59e0154020 0x7f59e015a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:06.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.466+0000 7f59e6659640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f59e00d01c0 0x7f59e0071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:06.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.466+0000 7f59e6659640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e015ad50 con 0x7f59e0072220 2023-12-04T05:47:06.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.466+0000 7f59e6659640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e00ca230 con 0x7f59e00d01c0 2023-12-04T05:47:06.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.466+0000 7f59e6659640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e00c6770 con 0x7f59e0154020 2023-12-04T05:47:06.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.466+0000 7f59df7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f59e00d01c0 0x7f59e0071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:06.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.466+0000 7f59df7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f59e00d01c0 0x7f59e0071ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54674/0 (socket says 172.21.15.107:54674) 2023-12-04T05:47:06.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.466+0000 7f59df7fe640 1 -- 172.21.15.107:0/2280761539 learned_addr learned my addr 172.21.15.107:0/2280761539 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:06.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.466+0000 7f59dffff640 1 --2- 172.21.15.107:0/2280761539 >> v2:172.21.15.195:3300/0 conn(0x7f59e0154020 0x7f59e015a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:06.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.466+0000 7f59df7fe640 1 -- 172.21.15.107:0/2280761539 >> v2:172.21.15.195:3300/0 conn(0x7f59e0154020 msgr2=0x7f59e015a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:06.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.466+0000 7f59df7fe640 1 --2- 172.21.15.107:0/2280761539 >> v2:172.21.15.195:3300/0 conn(0x7f59e0154020 0x7f59e015a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:06.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.466+0000 7f59df7fe640 1 -- 172.21.15.107:0/2280761539 >> v1:172.21.15.195:6789/0 conn(0x7f59e0072220 legacy=0x7f59e00c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:06.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.466+0000 7f59df7fe640 1 -- 172.21.15.107:0/2280761539 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e015d710 con 0x7f59e00d01c0 2023-12-04T05:47:06.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.466+0000 7f59df7fe640 1 --2- 172.21.15.107:0/2280761539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59e00d01c0 0x7f59e0071ce0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f59e00d0150 tx=0x7f59d401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9080f1b340f71828 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:06.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59deffd640 1 -- 172.21.15.107:0/2280761539 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d40a89b0 con 0x7f59e00d01c0 2023-12-04T05:47:06.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59deffd640 1 -- 172.21.15.107:0/2280761539 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d40a8b10 con 0x7f59e00d01c0 2023-12-04T05:47:06.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59deffd640 1 -- 172.21.15.107:0/2280761539 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d40a8ea0 con 0x7f59e00d01c0 2023-12-04T05:47:06.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e6659640 1 -- 172.21.15.107:0/2280761539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59e00d01c0 msgr2=0x7f59e0071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:06.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e6659640 1 --2- 172.21.15.107:0/2280761539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59e00d01c0 0x7f59e0071ce0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f59e00d0150 tx=0x7f59d401ecb0 comp rx=0 tx=0).stop 2023-12-04T05:47:06.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e6659640 1 -- 172.21.15.107:0/2280761539 shutdown_connections 2023-12-04T05:47:06.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e6659640 1 --2- 172.21.15.107:0/2280761539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59e00d01c0 0x7f59e0071ce0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:06.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e6659640 1 --2- 172.21.15.107:0/2280761539 >> v2:172.21.15.195:3300/0 conn(0x7f59e0154020 0x7f59e015a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:06.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e6659640 1 -- 172.21.15.107:0/2280761539 >> 172.21.15.107:0/2280761539 conn(0x7f59e006c6c0 msgr2=0x7f59e00cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:06.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e6659640 1 -- 172.21.15.107:0/2280761539 shutdown_connections 2023-12-04T05:47:06.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e6659640 1 -- 172.21.15.107:0/2280761539 wait complete. 2023-12-04T05:47:06.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e6659640 1 Processor -- start 2023-12-04T05:47:06.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e6659640 1 -- start start 2023-12-04T05:47:06.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e6659640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f59e0072220 0x7f59e00c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:06.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e6659640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f59e00d01c0 0x7f59e00cdf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:06.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e6659640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59e0154020 0x7f59e01f3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:06.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e6659640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e00cf020 con 0x7f59e0154020 2023-12-04T05:47:06.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e6659640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e01f4020 con 0x7f59e0072220 2023-12-04T05:47:06.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e6659640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e01f3d40 con 0x7f59e00d01c0 2023-12-04T05:47:06.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e4bf6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59e0154020 0x7f59e01f3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:06.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e4bf6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59e0154020 0x7f59e01f3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54678/0 (socket says 172.21.15.107:54678) 2023-12-04T05:47:06.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e4bf6640 1 -- 172.21.15.107:0/556883053 learned_addr learned my addr 172.21.15.107:0/556883053 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:06.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59df7fe640 1 --2- 172.21.15.107:0/556883053 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f59e00d01c0 0x7f59e00cdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:06.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e4bf6640 1 -- 172.21.15.107:0/556883053 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f59e00d01c0 msgr2=0x7f59e00cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:06.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59dffff640 1 --2- 172.21.15.107:0/556883053 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f59e0072220 0x7f59e00c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:06.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e4bf6640 1 --2- 172.21.15.107:0/556883053 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f59e00d01c0 0x7f59e00cdf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:06.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e4bf6640 1 -- 172.21.15.107:0/556883053 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f59e0072220 msgr2=0x7f59e00c9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:06.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e4bf6640 1 --2- 172.21.15.107:0/556883053 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f59e0072220 0x7f59e00c9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:06.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e4bf6640 1 -- 172.21.15.107:0/556883053 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d40094d0 con 0x7f59e0154020 2023-12-04T05:47:06.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e4bf6640 1 --2- 172.21.15.107:0/556883053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59e0154020 0x7f59e01f3370 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f59d800c780 tx=0x7f59d800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:06.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59dcff9640 1 -- 172.21.15.107:0/556883053 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d8004b70 con 0x7f59e0154020 2023-12-04T05:47:06.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59dcff9640 1 -- 172.21.15.107:0/556883053 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d800ce90 con 0x7f59e0154020 2023-12-04T05:47:06.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e6659640 1 -- 172.21.15.107:0/556883053 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e00ce680 con 0x7f59e0154020 2023-12-04T05:47:06.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59dcff9640 1 -- 172.21.15.107:0/556883053 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d800a850 con 0x7f59e0154020 2023-12-04T05:47:06.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.470+0000 7f59e6659640 1 -- 172.21.15.107:0/556883053 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e00cec40 con 0x7f59e0154020 2023-12-04T05:47:06.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.474+0000 7f59e6659640 1 -- 172.21.15.107:0/556883053 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e00cf160 con 0x7f59e0154020 2023-12-04T05:47:06.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.474+0000 7f59dcff9640 1 -- 172.21.15.107:0/556883053 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f59d800a9b0 con 0x7f59e0154020 2023-12-04T05:47:06.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.474+0000 7f59dcff9640 1 --2- 172.21.15.107:0/556883053 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f59cc0668d0 0x7f59cc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:06.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.474+0000 7f59dffff640 1 --2- 172.21.15.107:0/556883053 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f59cc0668d0 0x7f59cc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:06.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.474+0000 7f59dcff9640 1 -- 172.21.15.107:0/556883053 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f59d80847f0 con 0x7f59e0154020 2023-12-04T05:47:06.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.474+0000 7f59dffff640 1 --2- 172.21.15.107:0/556883053 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f59cc0668d0 0x7f59cc068d90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f59d000b7c0 tx=0x7f59d0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:06.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.478+0000 7f59dcff9640 1 -- 172.21.15.107:0/556883053 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f59e00cf160 con 0x7f59e0154020 2023-12-04T05:47:06.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.650+0000 7f59e6659640 1 -- 172.21.15.107:0/556883053 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59e00cf160 con 0x7f59e0154020 2023-12-04T05:47:06.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.650+0000 7f59dcff9640 1 -- 172.21.15.107:0/556883053 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f59e00cf160 con 0x7f59e0154020 2023-12-04T05:47:06.655 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:06.655 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:06.655 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:06.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.654+0000 7f59e6659640 1 -- 172.21.15.107:0/556883053 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f59cc0668d0 msgr2=0x7f59cc068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:06.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.654+0000 7f59e6659640 1 --2- 172.21.15.107:0/556883053 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f59cc0668d0 0x7f59cc068d90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f59d000b7c0 tx=0x7f59d0008040 comp rx=0 tx=0).stop 2023-12-04T05:47:06.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.654+0000 7f59e6659640 1 -- 172.21.15.107:0/556883053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59e0154020 msgr2=0x7f59e01f3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:06.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.654+0000 7f59e6659640 1 --2- 172.21.15.107:0/556883053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59e0154020 0x7f59e01f3370 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f59d800c780 tx=0x7f59d800cc40 comp rx=0 tx=0).stop 2023-12-04T05:47:06.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.654+0000 7f59e6659640 1 -- 172.21.15.107:0/556883053 shutdown_connections 2023-12-04T05:47:06.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.654+0000 7f59e6659640 1 --2- 172.21.15.107:0/556883053 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f59cc0668d0 0x7f59cc068d90 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:06.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.654+0000 7f59e6659640 1 --2- 172.21.15.107:0/556883053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59e0154020 0x7f59e01f3370 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:06.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.654+0000 7f59e6659640 1 --2- 172.21.15.107:0/556883053 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f59e00d01c0 0x7f59e00cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:06.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.654+0000 7f59e6659640 1 --2- 172.21.15.107:0/556883053 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f59e0072220 0x7f59e00c9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:06.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.654+0000 7f59e6659640 1 -- 172.21.15.107:0/556883053 >> 172.21.15.107:0/556883053 conn(0x7f59e006c6c0 msgr2=0x7f59e01e3180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:06.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.654+0000 7f59e6659640 1 -- 172.21.15.107:0/556883053 shutdown_connections 2023-12-04T05:47:06.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:06.654+0000 7f59e6659640 1 -- 172.21.15.107:0/556883053 wait complete. 2023-12-04T05:47:06.672 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:47:07.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~86s 2023-12-04T05:47:07.721 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~200s 2023-12-04T05:47:07.721 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~241s 2023-12-04T05:47:08.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:08.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.802+0000 7f8395b86640 1 Processor -- start 2023-12-04T05:47:08.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.802+0000 7f8395b86640 1 -- start start 2023-12-04T05:47:08.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.802+0000 7f8395b86640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8390072110 0x7f8390158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.802+0000 7f8395b86640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f83900c56f0 0x7f8390140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.802+0000 7f8395b86640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8390158da0 con 0x7f83900724f0 2023-12-04T05:47:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.802+0000 7f8395b86640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83901583d0 con 0x7f83900c56f0 2023-12-04T05:47:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.802+0000 7f8395b86640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83900c6170 con 0x7f8390072110 2023-12-04T05:47:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.802+0000 7f838effd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f83900724f0 0x7f83900c7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40738/0 (socket says 172.21.15.107:40738) 2023-12-04T05:47:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.802+0000 7f838effd640 1 -- 172.21.15.107:0/1088526739 learned_addr learned my addr 172.21.15.107:0/1088526739 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:47:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.802+0000 7f838ffff640 1 --2- 172.21.15.107:0/1088526739 >> v2:172.21.15.107:3300/0 conn(0x7f83900c56f0 0x7f8390140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.802+0000 7f838e7fc640 1 -- 172.21.15.107:0/1088526739 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 646034463 0 0) 0x7f8384001120 con 0x7f83900724f0 2023-12-04T05:47:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.802+0000 7f838f7fe640 1 --2- 172.21.15.107:0/1088526739 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8390072110 0x7f8390158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.802+0000 7f838e7fc640 1 -- 172.21.15.107:0/1088526739 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8378003530 con 0x7f83900724f0 2023-12-04T05:47:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.802+0000 7f838ffff640 1 -- 172.21.15.107:0/1088526739 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8390072110 msgr2=0x7f8390158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.802+0000 7f838ffff640 1 --2- 172.21.15.107:0/1088526739 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8390072110 0x7f8390158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.802+0000 7f838ffff640 1 -- 172.21.15.107:0/1088526739 >> v1:172.21.15.107:6789/0 conn(0x7f83900724f0 legacy=0x7f83900c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.802+0000 7f838ffff640 1 -- 172.21.15.107:0/1088526739 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8390157da0 con 0x7f83900c56f0 2023-12-04T05:47:08.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f838ffff640 1 --2- 172.21.15.107:0/1088526739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83900c56f0 0x7f8390140830 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f8388009d90 tx=0x7f838801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9531d25f9f4a074 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:08.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f838e7fc640 1 -- 172.21.15.107:0/1088526739 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83880a5900 con 0x7f83900c56f0 2023-12-04T05:47:08.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f838e7fc640 1 -- 172.21.15.107:0/1088526739 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83880a5a60 con 0x7f83900c56f0 2023-12-04T05:47:08.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f838e7fc640 1 -- 172.21.15.107:0/1088526739 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83880a5df0 con 0x7f83900c56f0 2023-12-04T05:47:08.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f8395b86640 1 -- 172.21.15.107:0/1088526739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83900c56f0 msgr2=0x7f8390140830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:08.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f8395b86640 1 --2- 172.21.15.107:0/1088526739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83900c56f0 0x7f8390140830 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f8388009d90 tx=0x7f838801c950 comp rx=0 tx=0).stop 2023-12-04T05:47:08.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f8395b86640 1 -- 172.21.15.107:0/1088526739 shutdown_connections 2023-12-04T05:47:08.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f8395b86640 1 --2- 172.21.15.107:0/1088526739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83900c56f0 0x7f8390140830 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:08.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f8395b86640 1 --2- 172.21.15.107:0/1088526739 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8390072110 0x7f8390158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:08.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f8395b86640 1 -- 172.21.15.107:0/1088526739 >> 172.21.15.107:0/1088526739 conn(0x7f83900c7ff0 msgr2=0x7f83900c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:08.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f8395b86640 1 -- 172.21.15.107:0/1088526739 shutdown_connections 2023-12-04T05:47:08.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f8395b86640 1 -- 172.21.15.107:0/1088526739 wait complete. 2023-12-04T05:47:08.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f8395b86640 1 Processor -- start 2023-12-04T05:47:08.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f8395b86640 1 -- start start 2023-12-04T05:47:08.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f8395b86640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8390072110 0x7f839013cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:08.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f8395b86640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f83900724f0 0x7f8390152d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f8395b86640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f83900c56f0 0x7f8390153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f8395b86640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839013dc90 con 0x7f8390072110 2023-12-04T05:47:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f8395b86640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83901546b0 con 0x7f83900724f0 2023-12-04T05:47:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f8395b86640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8390154990 con 0x7f83900c56f0 2023-12-04T05:47:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f838f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8390072110 0x7f839013cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f838f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8390072110 0x7f839013cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56574/0 (socket says 172.21.15.107:56574) 2023-12-04T05:47:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f838f7fe640 1 -- 172.21.15.107:0/1110538215 learned_addr learned my addr 172.21.15.107:0/1110538215 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f838f7fe640 1 -- 172.21.15.107:0/1110538215 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f83900c56f0 msgr2=0x7f8390153290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f838f7fe640 1 --2- 172.21.15.107:0/1110538215 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f83900c56f0 0x7f8390153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f838f7fe640 1 -- 172.21.15.107:0/1110538215 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f83900724f0 msgr2=0x7f8390152d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.806+0000 7f838effd640 1 --2- 172.21.15.107:0/1110538215 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f83900724f0 0x7f8390152d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.810+0000 7f838f7fe640 1 --2- 172.21.15.107:0/1110538215 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f83900724f0 0x7f8390152d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.810+0000 7f838f7fe640 1 -- 172.21.15.107:0/1110538215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8388009a70 con 0x7f8390072110 2023-12-04T05:47:08.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.810+0000 7f838f7fe640 1 --2- 172.21.15.107:0/1110538215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8390072110 0x7f839013cb80 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f838000d5f0 tx=0x7f838000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:08.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.810+0000 7f8377fff640 1 -- 172.21.15.107:0/1110538215 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83800125f0 con 0x7f8390072110 2023-12-04T05:47:08.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.810+0000 7f8377fff640 1 -- 172.21.15.107:0/1110538215 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8380012750 con 0x7f8390072110 2023-12-04T05:47:08.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.810+0000 7f8395b86640 1 -- 172.21.15.107:0/1110538215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8390154c70 con 0x7f8390072110 2023-12-04T05:47:08.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.810+0000 7f8377fff640 1 -- 172.21.15.107:0/1110538215 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8380012ae0 con 0x7f8390072110 2023-12-04T05:47:08.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.810+0000 7f8395b86640 1 -- 172.21.15.107:0/1110538215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83901541f0 con 0x7f8390072110 2023-12-04T05:47:08.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.810+0000 7f8395b86640 1 -- 172.21.15.107:0/1110538215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8390152690 con 0x7f8390072110 2023-12-04T05:47:08.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.810+0000 7f8377fff640 1 -- 172.21.15.107:0/1110538215 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f8380012c40 con 0x7f8390072110 2023-12-04T05:47:08.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.810+0000 7f8377fff640 1 --2- 172.21.15.107:0/1110538215 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f837c0668d0 0x7f837c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:08.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.810+0000 7f8377fff640 1 -- 172.21.15.107:0/1110538215 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f8380084be0 con 0x7f8390072110 2023-12-04T05:47:08.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.810+0000 7f838effd640 1 --2- 172.21.15.107:0/1110538215 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f837c0668d0 0x7f837c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:08.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.810+0000 7f838effd640 1 --2- 172.21.15.107:0/1110538215 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f837c0668d0 0x7f837c068d90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8390140d10 tx=0x7f8384009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:08.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.814+0000 7f8377fff640 1 -- 172.21.15.107:0/1110538215 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8390152690 con 0x7f8390072110 2023-12-04T05:47:08.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.994+0000 7f8395b86640 1 -- 172.21.15.107:0/1110538215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8390152690 con 0x7f8390072110 2023-12-04T05:47:08.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.998+0000 7f8377fff640 1 -- 172.21.15.107:0/1110538215 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f8390152690 con 0x7f8390072110 2023-12-04T05:47:08.999 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:08.999 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:08.999 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:09.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.998+0000 7f8395b86640 1 -- 172.21.15.107:0/1110538215 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f837c0668d0 msgr2=0x7f837c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:09.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.998+0000 7f8395b86640 1 --2- 172.21.15.107:0/1110538215 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f837c0668d0 0x7f837c068d90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8390140d10 tx=0x7f8384009040 comp rx=0 tx=0).stop 2023-12-04T05:47:09.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.998+0000 7f8395b86640 1 -- 172.21.15.107:0/1110538215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8390072110 msgr2=0x7f839013cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:09.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.998+0000 7f8395b86640 1 --2- 172.21.15.107:0/1110538215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8390072110 0x7f839013cb80 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f838000d5f0 tx=0x7f838000dab0 comp rx=0 tx=0).stop 2023-12-04T05:47:09.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.998+0000 7f8395b86640 1 -- 172.21.15.107:0/1110538215 shutdown_connections 2023-12-04T05:47:09.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.998+0000 7f8395b86640 1 --2- 172.21.15.107:0/1110538215 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f837c0668d0 0x7f837c068d90 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:09.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.998+0000 7f8395b86640 1 --2- 172.21.15.107:0/1110538215 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f83900c56f0 0x7f8390153290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:09.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.998+0000 7f8395b86640 1 --2- 172.21.15.107:0/1110538215 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f83900724f0 0x7f8390152d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:09.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.998+0000 7f8395b86640 1 --2- 172.21.15.107:0/1110538215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8390072110 0x7f839013cb80 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:09.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.998+0000 7f8395b86640 1 -- 172.21.15.107:0/1110538215 >> 172.21.15.107:0/1110538215 conn(0x7f83900c7ff0 msgr2=0x7f839006d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:09.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.998+0000 7f8395b86640 1 -- 172.21.15.107:0/1110538215 shutdown_connections 2023-12-04T05:47:09.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:08.998+0000 7f8395b86640 1 -- 172.21.15.107:0/1110538215 wait complete. 2023-12-04T05:47:09.017 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:47:11.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbba2867640 1 Processor -- start 2023-12-04T05:47:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbba2867640 1 -- start start 2023-12-04T05:47:11.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbba2867640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbb9c0d01c0 0x7fbb9c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:11.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbba2867640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fbb9c072220 0x7fbb9c0c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:11.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbba2867640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb9c15d710 con 0x7fbb9c154020 2023-12-04T05:47:11.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbba2867640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb9c0ca230 con 0x7fbb9c072220 2023-12-04T05:47:11.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbba2867640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9c0c6770 con 0x7fbb9c0d01c0 2023-12-04T05:47:11.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbb9bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbb9c154020 0x7fbb9c15a810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40752/0 (socket says 172.21.15.107:40752) 2023-12-04T05:47:11.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbb9bfff640 1 -- 172.21.15.107:0/1614506147 learned_addr learned my addr 172.21.15.107:0/1614506147 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:47:11.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbb93fff640 1 --2- 172.21.15.107:0/1614506147 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbb9c0d01c0 0x7fbb9c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:11.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbba0e04640 1 --2- 172.21.15.107:0/1614506147 >> v2:172.21.15.195:3300/0 conn(0x7fbb9c072220 0x7fbb9c0c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:11.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbb9b7fe640 1 -- 172.21.15.107:0/1614506147 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2863810664 0 0) 0x7fbb9c15d710 con 0x7fbb9c154020 2023-12-04T05:47:11.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbb9b7fe640 1 -- 172.21.15.107:0/1614506147 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb88003530 con 0x7fbb9c154020 2023-12-04T05:47:11.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbb9b7fe640 1 -- 172.21.15.107:0/1614506147 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4024909661 0 0) 0x7fbb88003530 con 0x7fbb9c154020 2023-12-04T05:47:11.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbb9b7fe640 1 -- 172.21.15.107:0/1614506147 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbb9c0d01c0 msgr2=0x7fbb9c071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:11.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbb9b7fe640 1 --2- 172.21.15.107:0/1614506147 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbb9c0d01c0 0x7fbb9c071ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:11.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbb9b7fe640 1 -- 172.21.15.107:0/1614506147 >> v2:172.21.15.195:3300/0 conn(0x7fbb9c072220 msgr2=0x7fbb9c0c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbb9b7fe640 1 --2- 172.21.15.107:0/1614506147 >> v2:172.21.15.195:3300/0 conn(0x7fbb9c072220 0x7fbb9c0c4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbb9b7fe640 1 -- 172.21.15.107:0/1614506147 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9c15d9f0 con 0x7fbb9c154020 2023-12-04T05:47:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbb9b7fe640 1 -- 172.21.15.107:0/1614506147 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7fbb8c001680 con 0x7fbb9c154020 2023-12-04T05:47:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbb9b7fe640 1 -- 172.21.15.107:0/1614506147 >> v1:172.21.15.107:6789/0 conn(0x7fbb9c154020 legacy=0x7fbb9c15a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbb9b7fe640 1 --2- 172.21.15.107:0/1614506147 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbb9c0d01c0 0x7fbb880039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbb9b7fe640 1 --2- 172.21.15.107:0/1614506147 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbb88003f30 0x7fbb880047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbb9b7fe640 1 --2- 172.21.15.107:0/1614506147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb88004d10 0x7fbb88005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbb9b7fe640 1 -- 172.21.15.107:0/1614506147 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8800ba20 con 0x7fbb88004d10 2023-12-04T05:47:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbb9b7fe640 1 -- 172.21.15.107:0/1614506147 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8800bb60 con 0x7fbb88003f30 2023-12-04T05:47:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbb9b7fe640 1 -- 172.21.15.107:0/1614506147 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8800bd20 con 0x7fbb9c0d01c0 2023-12-04T05:47:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbba0e04640 1 --2- 172.21.15.107:0/1614506147 >> v2:172.21.15.195:3300/0 conn(0x7fbb9c072220 0x7fbb9c0c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:47:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbba0e04640 1 --2- 172.21.15.107:0/1614506147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb88004d10 0x7fbb88005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:11.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbb93fff640 1 --2- 172.21.15.107:0/1614506147 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbb88003f30 0x7fbb880047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:11.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbb9bfff640 1 --2- 172.21.15.107:0/1614506147 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbb9c0d01c0 0x7fbb880039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:11.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbba0e04640 1 -- 172.21.15.107:0/1614506147 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbb9c0d01c0 msgr2=0x7fbb880039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:11.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbba0e04640 1 --2- 172.21.15.107:0/1614506147 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbb9c0d01c0 0x7fbb880039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:11.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbba0e04640 1 -- 172.21.15.107:0/1614506147 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbb88003f30 msgr2=0x7fbb880047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:11.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbba0e04640 1 --2- 172.21.15.107:0/1614506147 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbb88003f30 0x7fbb880047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:11.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.146+0000 7fbba0e04640 1 -- 172.21.15.107:0/1614506147 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb88003530 con 0x7fbb88004d10 2023-12-04T05:47:11.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.150+0000 7fbba0e04640 1 --2- 172.21.15.107:0/1614506147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb88004d10 0x7fbb88005190 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fbb940073e0 tx=0x7fbb9401cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4199065a447d2de6 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:11.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.150+0000 7fbb9b7fe640 1 -- 172.21.15.107:0/1614506147 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb940a5e20 con 0x7fbb88004d10 2023-12-04T05:47:11.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.150+0000 7fbb9b7fe640 1 -- 172.21.15.107:0/1614506147 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb940ad440 con 0x7fbb88004d10 2023-12-04T05:47:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.150+0000 7fbb9b7fe640 1 -- 172.21.15.107:0/1614506147 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb940ad7b0 con 0x7fbb88004d10 2023-12-04T05:47:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.150+0000 7fbba2867640 1 -- 172.21.15.107:0/1614506147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb88004d10 msgr2=0x7fbb88005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.150+0000 7fbba2867640 1 --2- 172.21.15.107:0/1614506147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb88004d10 0x7fbb88005190 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fbb940073e0 tx=0x7fbb9401cf20 comp rx=0 tx=0).stop 2023-12-04T05:47:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.150+0000 7fbba0e04640 1 -- 172.21.15.107:0/1614506147 reap_dead start 2023-12-04T05:47:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.150+0000 7fbba2867640 1 -- 172.21.15.107:0/1614506147 shutdown_connections 2023-12-04T05:47:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.150+0000 7fbba2867640 1 -- 172.21.15.107:0/1614506147 >> 172.21.15.107:0/1614506147 conn(0x7fbb9c06c6c0 msgr2=0x7fbb9c0cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.150+0000 7fbba2867640 1 -- 172.21.15.107:0/1614506147 shutdown_connections 2023-12-04T05:47:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.150+0000 7fbba2867640 1 -- 172.21.15.107:0/1614506147 wait complete. 2023-12-04T05:47:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.150+0000 7fbba2867640 1 Processor -- start 2023-12-04T05:47:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.150+0000 7fbba2867640 1 -- start start 2023-12-04T05:47:11.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.150+0000 7fbba2867640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbb9c150870 0x7fbb9c0cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:11.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.150+0000 7fbba2867640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbb9c150c50 0x7fbb9c1eefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:11.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.150+0000 7fbba2867640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb9c149a60 0x7fbb9c1f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:11.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.150+0000 7fbba2867640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9c0ca7f0 con 0x7fbb9c149a60 2023-12-04T05:47:11.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.150+0000 7fbba2867640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9c1f2c00 con 0x7fbb9c150c50 2023-12-04T05:47:11.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.150+0000 7fbba2867640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9c0ca9e0 con 0x7fbb9c150870 2023-12-04T05:47:11.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.150+0000 7fbba0e04640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb9c149a60 0x7fbb9c1f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:11.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.150+0000 7fbba0e04640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb9c149a60 0x7fbb9c1f3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56594/0 (socket says 172.21.15.107:56594) 2023-12-04T05:47:11.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.150+0000 7fbba0e04640 1 -- 172.21.15.107:0/3889784738 learned_addr learned my addr 172.21.15.107:0/3889784738 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:11.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.154+0000 7fbb9bfff640 1 --2- 172.21.15.107:0/3889784738 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbb9c150870 0x7fbb9c0cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:11.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.154+0000 7fbba0e04640 1 -- 172.21.15.107:0/3889784738 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbb9c150870 msgr2=0x7fbb9c0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:11.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.154+0000 7fbb93fff640 1 --2- 172.21.15.107:0/3889784738 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbb9c150c50 0x7fbb9c1eefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:11.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.154+0000 7fbba0e04640 1 --2- 172.21.15.107:0/3889784738 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbb9c150870 0x7fbb9c0cc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:11.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.154+0000 7fbba0e04640 1 -- 172.21.15.107:0/3889784738 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbb9c150c50 msgr2=0x7fbb9c1eefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:11.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.154+0000 7fbba0e04640 1 --2- 172.21.15.107:0/3889784738 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbb9c150c50 0x7fbb9c1eefc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:11.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.154+0000 7fbba0e04640 1 -- 172.21.15.107:0/3889784738 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9c0cacc0 con 0x7fbb9c149a60 2023-12-04T05:47:11.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.154+0000 7fbba0e04640 1 --2- 172.21.15.107:0/3889784738 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb9c149a60 0x7fbb9c1f3090 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fbb940b0040 tx=0x7fbb88005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:11.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.154+0000 7fbb997fa640 1 -- 172.21.15.107:0/3889784738 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb94009690 con 0x7fbb9c149a60 2023-12-04T05:47:11.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.154+0000 7fbb997fa640 1 -- 172.21.15.107:0/3889784738 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb940ada70 con 0x7fbb9c149a60 2023-12-04T05:47:11.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.154+0000 7fbba2867640 1 -- 172.21.15.107:0/3889784738 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb9c1f25b0 con 0x7fbb9c149a60 2023-12-04T05:47:11.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.154+0000 7fbb997fa640 1 -- 172.21.15.107:0/3889784738 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb940ade20 con 0x7fbb9c149a60 2023-12-04T05:47:11.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.154+0000 7fbba2867640 1 -- 172.21.15.107:0/3889784738 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb9c0cb1b0 con 0x7fbb9c149a60 2023-12-04T05:47:11.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.154+0000 7fbba2867640 1 -- 172.21.15.107:0/3889784738 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb9c067870 con 0x7fbb9c149a60 2023-12-04T05:47:11.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.154+0000 7fbb997fa640 1 -- 172.21.15.107:0/3889784738 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fbb940b4040 con 0x7fbb9c149a60 2023-12-04T05:47:11.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.154+0000 7fbb997fa640 1 --2- 172.21.15.107:0/3889784738 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbb7c066900 0x7fbb7c068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:11.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.154+0000 7fbb997fa640 1 -- 172.21.15.107:0/3889784738 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fbb9411ee50 con 0x7fbb9c149a60 2023-12-04T05:47:11.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.154+0000 7fbb9bfff640 1 --2- 172.21.15.107:0/3889784738 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbb7c066900 0x7fbb7c068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:11.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.154+0000 7fbb9bfff640 1 --2- 172.21.15.107:0/3889784738 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbb7c066900 0x7fbb7c068dc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c001060 tx=0x7fbb8c001160 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:11.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.158+0000 7fbb997fa640 1 -- 172.21.15.107:0/3889784738 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbb940ebec0 con 0x7fbb9c149a60 2023-12-04T05:47:11.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.338+0000 7fbba2867640 1 -- 172.21.15.107:0/3889784738 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb9c1f2d40 con 0x7fbb9c149a60 2023-12-04T05:47:11.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.338+0000 7fbb997fa640 1 -- 172.21.15.107:0/3889784738 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7fbb940b2070 con 0x7fbb9c149a60 2023-12-04T05:47:11.341 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:11.341 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:11.342 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:11.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.342+0000 7fbba2867640 1 -- 172.21.15.107:0/3889784738 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbb7c066900 msgr2=0x7fbb7c068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:11.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.342+0000 7fbba2867640 1 --2- 172.21.15.107:0/3889784738 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbb7c066900 0x7fbb7c068dc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c001060 tx=0x7fbb8c001160 comp rx=0 tx=0).stop 2023-12-04T05:47:11.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.342+0000 7fbba2867640 1 -- 172.21.15.107:0/3889784738 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb9c149a60 msgr2=0x7fbb9c1f3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:11.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.342+0000 7fbba2867640 1 --2- 172.21.15.107:0/3889784738 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb9c149a60 0x7fbb9c1f3090 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fbb940b0040 tx=0x7fbb88005800 comp rx=0 tx=0).stop 2023-12-04T05:47:11.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.342+0000 7fbba2867640 1 -- 172.21.15.107:0/3889784738 shutdown_connections 2023-12-04T05:47:11.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.342+0000 7fbba2867640 1 --2- 172.21.15.107:0/3889784738 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fbb7c066900 0x7fbb7c068dc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:11.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.342+0000 7fbba2867640 1 --2- 172.21.15.107:0/3889784738 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb9c149a60 0x7fbb9c1f3090 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:11.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.342+0000 7fbba2867640 1 --2- 172.21.15.107:0/3889784738 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbb9c150c50 0x7fbb9c1eefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:11.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.342+0000 7fbba2867640 1 --2- 172.21.15.107:0/3889784738 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fbb9c150870 0x7fbb9c0cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:11.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.342+0000 7fbba2867640 1 -- 172.21.15.107:0/3889784738 >> 172.21.15.107:0/3889784738 conn(0x7fbb9c06c6c0 msgr2=0x7fbb9c14fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:11.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.342+0000 7fbba2867640 1 -- 172.21.15.107:0/3889784738 shutdown_connections 2023-12-04T05:47:11.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:11.342+0000 7fbba2867640 1 -- 172.21.15.107:0/3889784738 wait complete. 2023-12-04T05:47:11.359 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:47:13.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:13.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.486+0000 7f4fb88a3640 1 Processor -- start 2023-12-04T05:47:13.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.490+0000 7f4fb88a3640 1 -- start start 2023-12-04T05:47:13.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.490+0000 7f4fb88a3640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4fb4154020 0x7f4fb415a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:13.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.490+0000 7f4fb88a3640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4fb40d01c0 0x7f4fb4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:13.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.490+0000 7f4fb88a3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4fb4072220 0x7f4fb40c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:13.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.490+0000 7f4fb88a3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb40ca230 con 0x7f4fb4072220 2023-12-04T05:47:13.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.490+0000 7f4fb88a3640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb40c6770 con 0x7f4fb4154020 2023-12-04T05:47:13.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.490+0000 7f4fb88a3640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb415ad50 con 0x7f4fb40d01c0 2023-12-04T05:47:13.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.490+0000 7f4fb2d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4fb4072220 0x7f4fb40c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:13.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.490+0000 7f4fb2d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4fb4072220 0x7f4fb40c4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56610/0 (socket says 172.21.15.107:56610) 2023-12-04T05:47:13.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.490+0000 7f4fb2d9d640 1 -- 172.21.15.107:0/2153513157 learned_addr learned my addr 172.21.15.107:0/2153513157 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:13.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.490+0000 7f4fb1d9b640 1 --2- 172.21.15.107:0/2153513157 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4fb40d01c0 0x7f4fb4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:13.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.490+0000 7f4fb259c640 1 --2- 172.21.15.107:0/2153513157 >> v2:172.21.15.195:3300/0 conn(0x7f4fb4154020 0x7f4fb415a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:13.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.490+0000 7f4fb2d9d640 1 -- 172.21.15.107:0/2153513157 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4fb40d01c0 msgr2=0x7f4fb4071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:13.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.490+0000 7f4fb2d9d640 1 --2- 172.21.15.107:0/2153513157 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4fb40d01c0 0x7f4fb4071ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:13.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.490+0000 7f4fb2d9d640 1 -- 172.21.15.107:0/2153513157 >> v2:172.21.15.195:3300/0 conn(0x7f4fb4154020 msgr2=0x7f4fb415a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:13.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.490+0000 7f4fb2d9d640 1 --2- 172.21.15.107:0/2153513157 >> v2:172.21.15.195:3300/0 conn(0x7f4fb4154020 0x7f4fb415a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:13.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.490+0000 7f4fb2d9d640 1 -- 172.21.15.107:0/2153513157 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb415d710 con 0x7f4fb4072220 2023-12-04T05:47:13.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.490+0000 7f4fb2d9d640 1 --2- 172.21.15.107:0/2153513157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fb4072220 0x7f4fb40c4c10 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f4fb40d0150 tx=0x7f4fac01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3c6a5de9e89c9a server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:13.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb159a640 1 -- 172.21.15.107:0/2153513157 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fac0af400 con 0x7f4fb4072220 2023-12-04T05:47:13.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb159a640 1 -- 172.21.15.107:0/2153513157 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fac0ab050 con 0x7f4fb4072220 2023-12-04T05:47:13.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb159a640 1 -- 172.21.15.107:0/2153513157 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fac0af780 con 0x7f4fb4072220 2023-12-04T05:47:13.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb88a3640 1 -- 172.21.15.107:0/2153513157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fb4072220 msgr2=0x7f4fb40c4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:13.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb88a3640 1 --2- 172.21.15.107:0/2153513157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fb4072220 0x7f4fb40c4c10 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f4fb40d0150 tx=0x7f4fac01f1c0 comp rx=0 tx=0).stop 2023-12-04T05:47:13.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb88a3640 1 -- 172.21.15.107:0/2153513157 shutdown_connections 2023-12-04T05:47:13.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb88a3640 1 --2- 172.21.15.107:0/2153513157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fb4072220 0x7f4fb40c4c10 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:13.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb88a3640 1 --2- 172.21.15.107:0/2153513157 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4fb40d01c0 0x7f4fb4071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:13.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb88a3640 1 --2- 172.21.15.107:0/2153513157 >> v2:172.21.15.195:3300/0 conn(0x7f4fb4154020 0x7f4fb415a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:13.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb88a3640 1 -- 172.21.15.107:0/2153513157 >> 172.21.15.107:0/2153513157 conn(0x7f4fb406c6c0 msgr2=0x7f4fb40cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:13.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb88a3640 1 -- 172.21.15.107:0/2153513157 shutdown_connections 2023-12-04T05:47:13.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb88a3640 1 -- 172.21.15.107:0/2153513157 wait complete. 2023-12-04T05:47:13.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb88a3640 1 Processor -- start 2023-12-04T05:47:13.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb88a3640 1 -- start start 2023-12-04T05:47:13.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb88a3640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4fb4072220 0x7f4fb40b3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:13.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb88a3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fb40d01c0 0x7f4fb40b3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:13.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb88a3640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4fb4154020 0x7f4fb4158c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:13.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb88a3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb40b3a40 con 0x7f4fb40d01c0 2023-12-04T05:47:13.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb88a3640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb40b4140 con 0x7f4fb4072220 2023-12-04T05:47:13.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb88a3640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb40c7330 con 0x7f4fb4154020 2023-12-04T05:47:13.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fb40d01c0 0x7f4fb40b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:13.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fb40d01c0 0x7f4fb40b3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56626/0 (socket says 172.21.15.107:56626) 2023-12-04T05:47:13.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb1d9b640 1 -- 172.21.15.107:0/3247313852 learned_addr learned my addr 172.21.15.107:0/3247313852 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:13.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb2d9d640 1 --2- 172.21.15.107:0/3247313852 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4fb4154020 0x7f4fb4158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:13.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb259c640 1 --2- 172.21.15.107:0/3247313852 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4fb4072220 0x7f4fb40b3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:13.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb1d9b640 1 -- 172.21.15.107:0/3247313852 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4fb4154020 msgr2=0x7f4fb4158c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:13.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb1d9b640 1 --2- 172.21.15.107:0/3247313852 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4fb4154020 0x7f4fb4158c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:13.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb1d9b640 1 -- 172.21.15.107:0/3247313852 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4fb4072220 msgr2=0x7f4fb40b3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:13.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb1d9b640 1 --2- 172.21.15.107:0/3247313852 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4fb4072220 0x7f4fb40b3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:13.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb1d9b640 1 -- 172.21.15.107:0/3247313852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fac009b30 con 0x7f4fb40d01c0 2023-12-04T05:47:13.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.494+0000 7f4fb2d9d640 1 --2- 172.21.15.107:0/3247313852 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4fb4154020 0x7f4fb4158c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:47:13.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.498+0000 7f4fb1d9b640 1 --2- 172.21.15.107:0/3247313852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fb40d01c0 0x7f4fb40b3c00 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4fa800c7f0 tx=0x7f4fa800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:13.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.498+0000 7f4fa2ffd640 1 -- 172.21.15.107:0/3247313852 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa8004220 con 0x7f4fb40d01c0 2023-12-04T05:47:13.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.498+0000 7f4fa2ffd640 1 -- 172.21.15.107:0/3247313852 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fa8004380 con 0x7f4fb40d01c0 2023-12-04T05:47:13.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.498+0000 7f4fb88a3640 1 -- 172.21.15.107:0/3247313852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb41f14a0 con 0x7f4fb40d01c0 2023-12-04T05:47:13.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.498+0000 7f4fa2ffd640 1 -- 172.21.15.107:0/3247313852 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa8003860 con 0x7f4fb40d01c0 2023-12-04T05:47:13.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.498+0000 7f4fb88a3640 1 -- 172.21.15.107:0/3247313852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb41f1a20 con 0x7f4fb40d01c0 2023-12-04T05:47:13.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.498+0000 7f4fb88a3640 1 -- 172.21.15.107:0/3247313852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fb4067870 con 0x7f4fb40d01c0 2023-12-04T05:47:13.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.498+0000 7f4fa2ffd640 1 -- 172.21.15.107:0/3247313852 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f4fa8003bd0 con 0x7f4fb40d01c0 2023-12-04T05:47:13.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.498+0000 7f4fa2ffd640 1 --2- 172.21.15.107:0/3247313852 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4f940668d0 0x7f4f94068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:13.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.498+0000 7f4fa2ffd640 1 -- 172.21.15.107:0/3247313852 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f4fa8084620 con 0x7f4fb40d01c0 2023-12-04T05:47:13.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.498+0000 7f4fb259c640 1 --2- 172.21.15.107:0/3247313852 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4f940668d0 0x7f4f94068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:13.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.498+0000 7f4fb259c640 1 --2- 172.21.15.107:0/3247313852 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4f940668d0 0x7f4f94068d90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400b9d0 tx=0x7f4fa4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:13.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.502+0000 7f4fa2ffd640 1 -- 172.21.15.107:0/3247313852 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4fa8051570 con 0x7f4fb40d01c0 2023-12-04T05:47:13.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.690+0000 7f4fb88a3640 1 -- 172.21.15.107:0/3247313852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4fb4072000 con 0x7f4fb40d01c0 2023-12-04T05:47:13.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.690+0000 7f4fa2ffd640 1 -- 172.21.15.107:0/3247313852 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f4fa801e090 con 0x7f4fb40d01c0 2023-12-04T05:47:13.693 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:13.693 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:13.694 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:13.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.694+0000 7f4fb88a3640 1 -- 172.21.15.107:0/3247313852 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4f940668d0 msgr2=0x7f4f94068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:13.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.694+0000 7f4fb88a3640 1 --2- 172.21.15.107:0/3247313852 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4f940668d0 0x7f4f94068d90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400b9d0 tx=0x7f4fa4008040 comp rx=0 tx=0).stop 2023-12-04T05:47:13.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.694+0000 7f4fb88a3640 1 -- 172.21.15.107:0/3247313852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fb40d01c0 msgr2=0x7f4fb40b3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:13.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.694+0000 7f4fb88a3640 1 --2- 172.21.15.107:0/3247313852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fb40d01c0 0x7f4fb40b3c00 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4fa800c7f0 tx=0x7f4fa800ccb0 comp rx=0 tx=0).stop 2023-12-04T05:47:13.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.694+0000 7f4fb88a3640 1 -- 172.21.15.107:0/3247313852 shutdown_connections 2023-12-04T05:47:13.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.694+0000 7f4fb88a3640 1 --2- 172.21.15.107:0/3247313852 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4f940668d0 0x7f4f94068d90 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:13.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.694+0000 7f4fb88a3640 1 --2- 172.21.15.107:0/3247313852 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f4fb4154020 0x7f4fb4158c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:13.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.694+0000 7f4fb88a3640 1 --2- 172.21.15.107:0/3247313852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fb40d01c0 0x7f4fb40b3c00 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:13.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.694+0000 7f4fb88a3640 1 --2- 172.21.15.107:0/3247313852 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4fb4072220 0x7f4fb40b3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:13.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.694+0000 7f4fb88a3640 1 -- 172.21.15.107:0/3247313852 >> 172.21.15.107:0/3247313852 conn(0x7f4fb406c6c0 msgr2=0x7f4fb4151860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:13.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.694+0000 7f4fb88a3640 1 -- 172.21.15.107:0/3247313852 shutdown_connections 2023-12-04T05:47:13.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:13.694+0000 7f4fb88a3640 1 -- 172.21.15.107:0/3247313852 wait complete. 2023-12-04T05:47:13.712 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:47:14.338 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~93s 2023-12-04T05:47:14.338 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~207s 2023-12-04T05:47:14.338 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~248s 2023-12-04T05:47:15.713 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:15.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.838+0000 7fef2b17f640 1 Processor -- start 2023-12-04T05:47:15.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.838+0000 7fef2b17f640 1 -- start start 2023-12-04T05:47:15.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.842+0000 7fef2b17f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fef240c5ac0 0x7fef2406a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:15.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.842+0000 7fef2b17f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fef2406aaf0 0x7fef24156410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:15.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.842+0000 7fef2b17f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef240c6990 con 0x7fef24070470 2023-12-04T05:47:15.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.842+0000 7fef2b17f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef240c7240 con 0x7fef240c5ac0 2023-12-04T05:47:15.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.842+0000 7fef2b17f640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef24072110 con 0x7fef2406aaf0 2023-12-04T05:47:15.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.842+0000 7fef28f1b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fef240c5ac0 0x7fef2406a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:15.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.842+0000 7fef28f1b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fef240c5ac0 0x7fef2406a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56638/0 (socket says 172.21.15.107:56638) 2023-12-04T05:47:15.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.842+0000 7fef28f1b640 1 -- 172.21.15.107:0/352107520 learned_addr learned my addr 172.21.15.107:0/352107520 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:15.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.842+0000 7fef23fff640 1 --2- 172.21.15.107:0/352107520 >> v2:172.21.15.195:3300/0 conn(0x7fef2406aaf0 0x7fef24156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:15.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.842+0000 7fef28f1b640 1 -- 172.21.15.107:0/352107520 >> v2:172.21.15.195:3300/0 conn(0x7fef2406aaf0 msgr2=0x7fef24156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:15.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.842+0000 7fef28f1b640 1 --2- 172.21.15.107:0/352107520 >> v2:172.21.15.195:3300/0 conn(0x7fef2406aaf0 0x7fef24156410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:15.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.842+0000 7fef28f1b640 1 -- 172.21.15.107:0/352107520 >> v1:172.21.15.195:6789/0 conn(0x7fef24070470 legacy=0x7fef24156950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:15.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.842+0000 7fef28f1b640 1 -- 172.21.15.107:0/352107520 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef240c6c70 con 0x7fef240c5ac0 2023-12-04T05:47:15.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.842+0000 7fef28f1b640 1 --2- 172.21.15.107:0/352107520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef240c5ac0 0x7fef2406a5b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fef2406c1b0 tx=0x7fef1801ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf1a6771c0c6515 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:15.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.842+0000 7fef237fe640 1 -- 172.21.15.107:0/352107520 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef180a89b0 con 0x7fef240c5ac0 2023-12-04T05:47:15.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.842+0000 7fef237fe640 1 -- 172.21.15.107:0/352107520 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef180a8b10 con 0x7fef240c5ac0 2023-12-04T05:47:15.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.842+0000 7fef237fe640 1 -- 172.21.15.107:0/352107520 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef180a8ea0 con 0x7fef240c5ac0 2023-12-04T05:47:15.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef2b17f640 1 -- 172.21.15.107:0/352107520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef240c5ac0 msgr2=0x7fef2406a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:15.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef2b17f640 1 --2- 172.21.15.107:0/352107520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef240c5ac0 0x7fef2406a5b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fef2406c1b0 tx=0x7fef1801ecb0 comp rx=0 tx=0).stop 2023-12-04T05:47:15.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef2b17f640 1 -- 172.21.15.107:0/352107520 shutdown_connections 2023-12-04T05:47:15.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef2b17f640 1 --2- 172.21.15.107:0/352107520 >> v2:172.21.15.195:3300/0 conn(0x7fef2406aaf0 0x7fef24156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:15.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef2b17f640 1 --2- 172.21.15.107:0/352107520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef240c5ac0 0x7fef2406a5b0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:15.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef2b17f640 1 -- 172.21.15.107:0/352107520 >> 172.21.15.107:0/352107520 conn(0x7fef2406e710 msgr2=0x7fef240c8580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:15.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef2b17f640 1 -- 172.21.15.107:0/352107520 shutdown_connections 2023-12-04T05:47:15.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef2b17f640 1 -- 172.21.15.107:0/352107520 wait complete. 2023-12-04T05:47:15.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef2b17f640 1 Processor -- start 2023-12-04T05:47:15.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef2b17f640 1 -- start start 2023-12-04T05:47:15.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef2b17f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef2406aaf0 0x7fef240bffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:15.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef2b17f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fef24070470 0x7fef24151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:15.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef2b17f640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef240c5ac0 0x7fef24159170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:15.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef2b17f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef24153710 con 0x7fef2406aaf0 2023-12-04T05:47:15.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef2b17f640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef24153150 con 0x7fef240c5ac0 2023-12-04T05:47:15.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef2b17f640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef24153430 con 0x7fef24070470 2023-12-04T05:47:15.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef28f1b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef2406aaf0 0x7fef240bffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:15.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef28f1b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef2406aaf0 0x7fef240bffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56644/0 (socket says 172.21.15.107:56644) 2023-12-04T05:47:15.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef28f1b640 1 -- 172.21.15.107:0/3260781257 learned_addr learned my addr 172.21.15.107:0/3260781257 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:15.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef28f1b640 1 -- 172.21.15.107:0/3260781257 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fef24070470 msgr2=0x7fef24151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:15.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef23fff640 1 --2- 172.21.15.107:0/3260781257 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fef24070470 0x7fef24151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef28f1b640 1 --2- 172.21.15.107:0/3260781257 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fef24070470 0x7fef24151a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef2971c640 1 --2- 172.21.15.107:0/3260781257 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef240c5ac0 0x7fef24159170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef28f1b640 1 -- 172.21.15.107:0/3260781257 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef240c5ac0 msgr2=0x7fef24159170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef28f1b640 1 --2- 172.21.15.107:0/3260781257 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef240c5ac0 0x7fef24159170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef28f1b640 1 -- 172.21.15.107:0/3260781257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef180094d0 con 0x7fef2406aaf0 2023-12-04T05:47:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef28f1b640 1 --2- 172.21.15.107:0/3260781257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef2406aaf0 0x7fef240bffe0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fef180b3040 tx=0x7fef1801ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef217fa640 1 -- 172.21.15.107:0/3260781257 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef180b0e80 con 0x7fef2406aaf0 2023-12-04T05:47:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef217fa640 1 -- 172.21.15.107:0/3260781257 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef180bb480 con 0x7fef2406aaf0 2023-12-04T05:47:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef2b17f640 1 -- 172.21.15.107:0/3260781257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef24158970 con 0x7fef2406aaf0 2023-12-04T05:47:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef217fa640 1 -- 172.21.15.107:0/3260781257 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef180bb830 con 0x7fef2406aaf0 2023-12-04T05:47:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.846+0000 7fef2b17f640 1 -- 172.21.15.107:0/3260781257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef240bfb10 con 0x7fef2406aaf0 2023-12-04T05:47:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.850+0000 7fef2b17f640 1 -- 172.21.15.107:0/3260781257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef241587d0 con 0x7fef2406aaf0 2023-12-04T05:47:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.850+0000 7fef217fa640 1 -- 172.21.15.107:0/3260781257 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fef180bb990 con 0x7fef2406aaf0 2023-12-04T05:47:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.850+0000 7fef217fa640 1 --2- 172.21.15.107:0/3260781257 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fef080668d0 0x7fef08068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.850+0000 7fef217fa640 1 -- 172.21.15.107:0/3260781257 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fef18122d20 con 0x7fef2406aaf0 2023-12-04T05:47:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.850+0000 7fef23fff640 1 --2- 172.21.15.107:0/3260781257 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fef080668d0 0x7fef08068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.850+0000 7fef23fff640 1 --2- 172.21.15.107:0/3260781257 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fef080668d0 0x7fef08068d90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fef2415a240 tx=0x7fef10008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:15.854+0000 7fef217fa640 1 -- 172.21.15.107:0/3260781257 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fef241587d0 con 0x7fef2406aaf0 2023-12-04T05:47:16.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:16.034+0000 7fef2b17f640 1 -- 172.21.15.107:0/3260781257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef24158e90 con 0x7fef2406aaf0 2023-12-04T05:47:16.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:16.034+0000 7fef217fa640 1 -- 172.21.15.107:0/3260781257 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7fef24158e90 con 0x7fef2406aaf0 2023-12-04T05:47:16.036 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:16.036 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:16.036 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:16.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:16.034+0000 7fef2b17f640 1 -- 172.21.15.107:0/3260781257 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fef080668d0 msgr2=0x7fef08068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:16.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:16.034+0000 7fef2b17f640 1 --2- 172.21.15.107:0/3260781257 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fef080668d0 0x7fef08068d90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fef2415a240 tx=0x7fef10008040 comp rx=0 tx=0).stop 2023-12-04T05:47:16.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:16.034+0000 7fef2b17f640 1 -- 172.21.15.107:0/3260781257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef2406aaf0 msgr2=0x7fef240bffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:16.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:16.034+0000 7fef2b17f640 1 --2- 172.21.15.107:0/3260781257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef2406aaf0 0x7fef240bffe0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fef180b3040 tx=0x7fef1801ba50 comp rx=0 tx=0).stop 2023-12-04T05:47:16.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:16.034+0000 7fef2b17f640 1 -- 172.21.15.107:0/3260781257 shutdown_connections 2023-12-04T05:47:16.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:16.034+0000 7fef2b17f640 1 --2- 172.21.15.107:0/3260781257 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fef080668d0 0x7fef08068d90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:16.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:16.034+0000 7fef2b17f640 1 --2- 172.21.15.107:0/3260781257 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fef240c5ac0 0x7fef24159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:16.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:16.034+0000 7fef2b17f640 1 --2- 172.21.15.107:0/3260781257 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fef24070470 0x7fef24151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:16.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:16.034+0000 7fef2b17f640 1 --2- 172.21.15.107:0/3260781257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef2406aaf0 0x7fef240bffe0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:16.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:16.034+0000 7fef2b17f640 1 -- 172.21.15.107:0/3260781257 >> 172.21.15.107:0/3260781257 conn(0x7fef2406e710 msgr2=0x7fef240c98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:16.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:16.034+0000 7fef2b17f640 1 -- 172.21.15.107:0/3260781257 shutdown_connections 2023-12-04T05:47:16.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:16.038+0000 7fef2b17f640 1 -- 172.21.15.107:0/3260781257 wait complete. 2023-12-04T05:47:16.054 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 50, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-12-04T05:39:59.032019+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_4288": { "addr": "172.21.15.195:6837/3412865477", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6835", "nonce": 3412865477, "type": "v2" }, { "addr": "172.21.15.195:6837", "nonce": 3412865477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_5190": { "addr": "172.21.15.107:6836/2928623624", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 2928623624, "type": "v2" }, { "addr": "172.21.15.107:6836", "nonce": 2928623624, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5190, "incarnation": 45, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:rejoin", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-12-04T05:46:11.040247+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2, 3, 4 ], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 5190 } } } ], "standbys": [ { "addr": "172.21.15.107:6837/2752497349", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6835", "nonce": 2752497349, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 2752497349, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 49, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5292, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-12-04T05:47:18.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c2f96b640 1 Processor -- start 2023-12-04T05:47:18.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c2f96b640 1 -- start start 2023-12-04T05:47:18.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c2f96b640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c280724f0 0x7f3c28158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:18.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c2f96b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c28158da0 con 0x7f3c28072110 2023-12-04T05:47:18.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c2f96b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c281583d0 con 0x7f3c280c56f0 2023-12-04T05:47:18.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c2f96b640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c280c6170 con 0x7f3c280724f0 2023-12-04T05:47:18.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c2d707640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3c28072110 0x7f3c280c7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54504/0 (socket says 172.21.15.107:54504) 2023-12-04T05:47:18.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c2d707640 1 -- 172.21.15.107:0/3622266915 learned_addr learned my addr 172.21.15.107:0/3622266915 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:47:18.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c2cf06640 1 --2- 172.21.15.107:0/3622266915 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c280724f0 0x7f3c28158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:18.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c1bfff640 1 -- 172.21.15.107:0/3622266915 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470300904 0 0) 0x7f3c24001120 con 0x7f3c28072110 2023-12-04T05:47:18.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c1bfff640 1 -- 172.21.15.107:0/3622266915 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c14003530 con 0x7f3c28072110 2023-12-04T05:47:18.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c1bfff640 1 -- 172.21.15.107:0/3622266915 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3784106594 0 0) 0x7f3c14003530 con 0x7f3c28072110 2023-12-04T05:47:18.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c1bfff640 1 -- 172.21.15.107:0/3622266915 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c280724f0 msgr2=0x7f3c28158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:18.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c1bfff640 1 --2- 172.21.15.107:0/3622266915 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c280724f0 0x7f3c28158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:18.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c1bfff640 1 -- 172.21.15.107:0/3622266915 >> v1:172.21.15.195:6789/0 conn(0x7f3c280c56f0 legacy=0x7f3c28140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:18.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c1bfff640 1 -- 172.21.15.107:0/3622266915 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c28157da0 con 0x7f3c28072110 2023-12-04T05:47:18.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c1bfff640 1 -- 172.21.15.107:0/3622266915 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f3c24001810 con 0x7f3c28072110 2023-12-04T05:47:18.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c1bfff640 1 -- 172.21.15.107:0/3622266915 >> v1:172.21.15.107:6789/0 conn(0x7f3c28072110 legacy=0x7f3c280c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:18.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c1bfff640 1 --2- 172.21.15.107:0/3622266915 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c140039b0 0x7f3c14003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:18.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c1bfff640 1 --2- 172.21.15.107:0/3622266915 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c140047d0 0x7f3c14004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:18.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c1bfff640 1 --2- 172.21.15.107:0/3622266915 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c280724f0 0x7f3c14005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:18.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c1bfff640 1 -- 172.21.15.107:0/3622266915 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c280c6170 con 0x7f3c140047d0 2023-12-04T05:47:18.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c1bfff640 1 -- 172.21.15.107:0/3622266915 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1400bb00 con 0x7f3c140039b0 2023-12-04T05:47:18.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c1bfff640 1 -- 172.21.15.107:0/3622266915 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1400bcc0 con 0x7f3c280724f0 2023-12-04T05:47:18.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c2cf06640 1 --2- 172.21.15.107:0/3622266915 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c140047d0 0x7f3c14004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:18.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c2d707640 1 --2- 172.21.15.107:0/3622266915 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c140039b0 0x7f3c14003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:18.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c2df08640 1 --2- 172.21.15.107:0/3622266915 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c280724f0 0x7f3c14005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:18.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c2cf06640 1 -- 172.21.15.107:0/3622266915 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c280724f0 msgr2=0x7f3c14005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:18.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c2cf06640 1 --2- 172.21.15.107:0/3622266915 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c280724f0 0x7f3c14005190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:18.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c2cf06640 1 -- 172.21.15.107:0/3622266915 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c140039b0 msgr2=0x7f3c14003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:18.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c2cf06640 1 --2- 172.21.15.107:0/3622266915 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c140039b0 0x7f3c14003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:18.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.182+0000 7f3c2cf06640 1 -- 172.21.15.107:0/3622266915 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c14003530 con 0x7f3c140047d0 2023-12-04T05:47:18.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2cf06640 1 --2- 172.21.15.107:0/3622266915 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c140047d0 0x7f3c14004c50 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c009ab0 tx=0x7f3c1c01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf3f42c1a7756371 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:18.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c1bfff640 1 -- 172.21.15.107:0/3622266915 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c0a5600 con 0x7f3c140047d0 2023-12-04T05:47:18.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c1bfff640 1 -- 172.21.15.107:0/3622266915 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c1c0a5760 con 0x7f3c140047d0 2023-12-04T05:47:18.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c1bfff640 1 -- 172.21.15.107:0/3622266915 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c0a5b10 con 0x7f3c140047d0 2023-12-04T05:47:18.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2f96b640 1 -- 172.21.15.107:0/3622266915 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c140047d0 msgr2=0x7f3c14004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:18.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2f96b640 1 --2- 172.21.15.107:0/3622266915 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c140047d0 0x7f3c14004c50 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c009ab0 tx=0x7f3c1c01ca10 comp rx=0 tx=0).stop 2023-12-04T05:47:18.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2df08640 1 -- 172.21.15.107:0/3622266915 reap_dead start 2023-12-04T05:47:18.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2f96b640 1 -- 172.21.15.107:0/3622266915 shutdown_connections 2023-12-04T05:47:18.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2f96b640 1 -- 172.21.15.107:0/3622266915 >> 172.21.15.107:0/3622266915 conn(0x7f3c280c7ff0 msgr2=0x7f3c280c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:18.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2f96b640 1 -- 172.21.15.107:0/3622266915 shutdown_connections 2023-12-04T05:47:18.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2f96b640 1 -- 172.21.15.107:0/3622266915 wait complete. 2023-12-04T05:47:18.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2f96b640 1 Processor -- start 2023-12-04T05:47:18.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2f96b640 1 -- start start 2023-12-04T05:47:18.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2f96b640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c140047d0 0x7f3c28152690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:18.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2f96b640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c28153870 0x7f3c28159cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:18.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2f96b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c2814c4b0 0x7f3c2815b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:18.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2f96b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2815a1f0 con 0x7f3c2814c4b0 2023-12-04T05:47:18.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2f96b640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c28159110 con 0x7f3c140047d0 2023-12-04T05:47:18.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2f96b640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2814ba60 con 0x7f3c28153870 2023-12-04T05:47:18.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2df08640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c2814c4b0 0x7f3c2815b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:18.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2df08640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c2814c4b0 0x7f3c2815b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41332/0 (socket says 172.21.15.107:41332) 2023-12-04T05:47:18.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2df08640 1 -- 172.21.15.107:0/2304948953 learned_addr learned my addr 172.21.15.107:0/2304948953 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:18.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2d707640 1 --2- 172.21.15.107:0/2304948953 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c140047d0 0x7f3c28152690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:18.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2cf06640 1 --2- 172.21.15.107:0/2304948953 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c28153870 0x7f3c28159cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:18.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2df08640 1 -- 172.21.15.107:0/2304948953 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c28153870 msgr2=0x7f3c28159cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:18.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2df08640 1 --2- 172.21.15.107:0/2304948953 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c28153870 0x7f3c28159cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:18.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2df08640 1 -- 172.21.15.107:0/2304948953 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c140047d0 msgr2=0x7f3c28152690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:18.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2df08640 1 --2- 172.21.15.107:0/2304948953 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c140047d0 0x7f3c28152690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:18.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.186+0000 7f3c2df08640 1 -- 172.21.15.107:0/2304948953 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c280ca640 con 0x7f3c2814c4b0 2023-12-04T05:47:18.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.190+0000 7f3c2df08640 1 --2- 172.21.15.107:0/2304948953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c2814c4b0 0x7f3c2815b8d0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3c280c5ad0 tx=0x7f3c2000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:18.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.190+0000 7f3c19ffb640 1 -- 172.21.15.107:0/2304948953 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c20003b80 con 0x7f3c2814c4b0 2023-12-04T05:47:18.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.190+0000 7f3c19ffb640 1 -- 172.21.15.107:0/2304948953 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c20003d00 con 0x7f3c2814c4b0 2023-12-04T05:47:18.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.190+0000 7f3c2f96b640 1 -- 172.21.15.107:0/2304948953 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c280ca920 con 0x7f3c2814c4b0 2023-12-04T05:47:18.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.190+0000 7f3c19ffb640 1 -- 172.21.15.107:0/2304948953 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c200052b0 con 0x7f3c2814c4b0 2023-12-04T05:47:18.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.190+0000 7f3c2f96b640 1 -- 172.21.15.107:0/2304948953 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c2814bcb0 con 0x7f3c2814c4b0 2023-12-04T05:47:18.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.190+0000 7f3c2f96b640 1 -- 172.21.15.107:0/2304948953 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c28067870 con 0x7f3c2814c4b0 2023-12-04T05:47:18.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.190+0000 7f3c19ffb640 1 -- 172.21.15.107:0/2304948953 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f3c20005410 con 0x7f3c2814c4b0 2023-12-04T05:47:18.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.190+0000 7f3c19ffb640 1 --2- 172.21.15.107:0/2304948953 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3c0c0668d0 0x7f3c0c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:18.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.190+0000 7f3c19ffb640 1 -- 172.21.15.107:0/2304948953 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f3c20083fc0 con 0x7f3c2814c4b0 2023-12-04T05:47:18.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.190+0000 7f3c2d707640 1 --2- 172.21.15.107:0/2304948953 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3c0c0668d0 0x7f3c0c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:18.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.190+0000 7f3c2d707640 1 --2- 172.21.15.107:0/2304948953 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3c0c0668d0 0x7f3c0c068d90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3c24000c10 tx=0x7f3c24009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:18.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.194+0000 7f3c19ffb640 1 -- 172.21.15.107:0/2304948953 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3c200b5280 con 0x7f3c2814c4b0 2023-12-04T05:47:18.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.374+0000 7f3c2f96b640 1 -- 172.21.15.107:0/2304948953 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c280cb2d0 con 0x7f3c2814c4b0 2023-12-04T05:47:18.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.374+0000 7f3c19ffb640 1 -- 172.21.15.107:0/2304948953 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f3c20018020 con 0x7f3c2814c4b0 2023-12-04T05:47:18.378 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:18.378 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:18.379 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:18.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.378+0000 7f3c2f96b640 1 -- 172.21.15.107:0/2304948953 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3c0c0668d0 msgr2=0x7f3c0c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:18.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.378+0000 7f3c2f96b640 1 --2- 172.21.15.107:0/2304948953 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3c0c0668d0 0x7f3c0c068d90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3c24000c10 tx=0x7f3c24009040 comp rx=0 tx=0).stop 2023-12-04T05:47:18.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.378+0000 7f3c2f96b640 1 -- 172.21.15.107:0/2304948953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c2814c4b0 msgr2=0x7f3c2815b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:18.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.378+0000 7f3c2f96b640 1 --2- 172.21.15.107:0/2304948953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c2814c4b0 0x7f3c2815b8d0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3c280c5ad0 tx=0x7f3c2000dc00 comp rx=0 tx=0).stop 2023-12-04T05:47:18.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.378+0000 7f3c2f96b640 1 -- 172.21.15.107:0/2304948953 shutdown_connections 2023-12-04T05:47:18.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.378+0000 7f3c2f96b640 1 --2- 172.21.15.107:0/2304948953 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f3c0c0668d0 0x7f3c0c068d90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:18.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.378+0000 7f3c2f96b640 1 --2- 172.21.15.107:0/2304948953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c2814c4b0 0x7f3c2815b8d0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:18.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.378+0000 7f3c2f96b640 1 --2- 172.21.15.107:0/2304948953 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f3c28153870 0x7f3c28159cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:18.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.378+0000 7f3c2f96b640 1 --2- 172.21.15.107:0/2304948953 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3c140047d0 0x7f3c28152690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:18.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.378+0000 7f3c2f96b640 1 -- 172.21.15.107:0/2304948953 >> 172.21.15.107:0/2304948953 conn(0x7f3c280c7ff0 msgr2=0x7f3c28155c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:18.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.378+0000 7f3c2f96b640 1 -- 172.21.15.107:0/2304948953 shutdown_connections 2023-12-04T05:47:18.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:18.378+0000 7f3c2f96b640 1 -- 172.21.15.107:0/2304948953 wait complete. 2023-12-04T05:47:18.396 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:47:20.397 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:20.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7e7476640 1 Processor -- start 2023-12-04T05:47:20.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7e7476640 1 -- start start 2023-12-04T05:47:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7e7476640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa7e00724f0 0x7fa7e0158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7e7476640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa7e00c56f0 0x7fa7e0140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7e7476640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e0158da0 con 0x7fa7e0072110 2023-12-04T05:47:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7e7476640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e01583d0 con 0x7fa7e00c56f0 2023-12-04T05:47:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7e7476640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e00c6170 con 0x7fa7e00724f0 2023-12-04T05:47:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7e5212640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa7e0072110 0x7fa7e00c7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54506/0 (socket says 172.21.15.107:54506) 2023-12-04T05:47:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7e5212640 1 -- 172.21.15.107:0/1660367161 learned_addr learned my addr 172.21.15.107:0/1660367161 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:47:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7d7fff640 1 -- 172.21.15.107:0/1660367161 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2070530347 0 0) 0x7fa7dc001120 con 0x7fa7e0072110 2023-12-04T05:47:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7d7fff640 1 -- 172.21.15.107:0/1660367161 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7cc003530 con 0x7fa7e0072110 2023-12-04T05:47:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7e4a11640 1 --2- 172.21.15.107:0/1660367161 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa7e00724f0 0x7fa7e0158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7e5a13640 1 --2- 172.21.15.107:0/1660367161 >> v2:172.21.15.195:3300/0 conn(0x7fa7e00c56f0 0x7fa7e0140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7d7fff640 1 -- 172.21.15.107:0/1660367161 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2509227868 0 0) 0x7fa7cc003530 con 0x7fa7e0072110 2023-12-04T05:47:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7d7fff640 1 -- 172.21.15.107:0/1660367161 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa7e00724f0 msgr2=0x7fa7e0158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7d7fff640 1 --2- 172.21.15.107:0/1660367161 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa7e00724f0 0x7fa7e0158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7d7fff640 1 -- 172.21.15.107:0/1660367161 >> v2:172.21.15.195:3300/0 conn(0x7fa7e00c56f0 msgr2=0x7fa7e0140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7d7fff640 1 --2- 172.21.15.107:0/1660367161 >> v2:172.21.15.195:3300/0 conn(0x7fa7e00c56f0 0x7fa7e0140830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7d7fff640 1 -- 172.21.15.107:0/1660367161 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e0157da0 con 0x7fa7e0072110 2023-12-04T05:47:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7d7fff640 1 -- 172.21.15.107:0/1660367161 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7fa7dc001810 con 0x7fa7e0072110 2023-12-04T05:47:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7e5a13640 1 --2- 172.21.15.107:0/1660367161 >> v2:172.21.15.195:3300/0 conn(0x7fa7e00c56f0 0x7fa7e0140830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:47:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7d7fff640 1 -- 172.21.15.107:0/1660367161 >> v1:172.21.15.107:6789/0 conn(0x7fa7e0072110 legacy=0x7fa7e00c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7d7fff640 1 --2- 172.21.15.107:0/1660367161 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7cc003980 0x7fa7cc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7d7fff640 1 --2- 172.21.15.107:0/1660367161 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa7e00724f0 0x7fa7cc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7d7fff640 1 --2- 172.21.15.107:0/1660367161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7cc004d10 0x7fa7cc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7d7fff640 1 -- 172.21.15.107:0/1660367161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e01583d0 con 0x7fa7cc004d10 2023-12-04T05:47:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7d7fff640 1 -- 172.21.15.107:0/1660367161 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e00c6170 con 0x7fa7cc003980 2023-12-04T05:47:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7d7fff640 1 -- 172.21.15.107:0/1660367161 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7cc00fe80 con 0x7fa7e00724f0 2023-12-04T05:47:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7e5a13640 1 --2- 172.21.15.107:0/1660367161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7cc004d10 0x7fa7cc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7e4a11640 1 --2- 172.21.15.107:0/1660367161 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa7e00724f0 0x7fa7cc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7e5212640 1 --2- 172.21.15.107:0/1660367161 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7cc003980 0x7fa7cc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7e5a13640 1 -- 172.21.15.107:0/1660367161 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa7e00724f0 msgr2=0x7fa7cc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7e5a13640 1 --2- 172.21.15.107:0/1660367161 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa7e00724f0 0x7fa7cc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7e5a13640 1 -- 172.21.15.107:0/1660367161 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7cc003980 msgr2=0x7fa7cc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:20.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7e5a13640 1 --2- 172.21.15.107:0/1660367161 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7cc003980 0x7fa7cc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:20.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7e5a13640 1 -- 172.21.15.107:0/1660367161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7cc003530 con 0x7fa7cc004d10 2023-12-04T05:47:20.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7e4a11640 1 --2- 172.21.15.107:0/1660367161 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa7e00724f0 0x7fa7cc0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:47:20.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.526+0000 7fa7e5212640 1 --2- 172.21.15.107:0/1660367161 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7cc003980 0x7fa7cc003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:47:20.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.530+0000 7fa7e5a13640 1 --2- 172.21.15.107:0/1660367161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7cc004d10 0x7fa7cc005190 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8007410 tx=0x7fa7d801cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce0e744cf5b77cf server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:20.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.530+0000 7fa7d7fff640 1 -- 172.21.15.107:0/1660367161 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d80a5e20 con 0x7fa7cc004d10 2023-12-04T05:47:20.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.530+0000 7fa7d7fff640 1 -- 172.21.15.107:0/1660367161 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d80ad440 con 0x7fa7cc004d10 2023-12-04T05:47:20.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.530+0000 7fa7d7fff640 1 -- 172.21.15.107:0/1660367161 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d80ad7b0 con 0x7fa7cc004d10 2023-12-04T05:47:20.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.530+0000 7fa7e7476640 1 -- 172.21.15.107:0/1660367161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7cc004d10 msgr2=0x7fa7cc005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:20.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.530+0000 7fa7e7476640 1 --2- 172.21.15.107:0/1660367161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7cc004d10 0x7fa7cc005190 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8007410 tx=0x7fa7d801cf20 comp rx=0 tx=0).stop 2023-12-04T05:47:20.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.530+0000 7fa7e5a13640 1 -- 172.21.15.107:0/1660367161 reap_dead start 2023-12-04T05:47:20.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.530+0000 7fa7e7476640 1 -- 172.21.15.107:0/1660367161 shutdown_connections 2023-12-04T05:47:20.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.530+0000 7fa7e7476640 1 -- 172.21.15.107:0/1660367161 >> 172.21.15.107:0/1660367161 conn(0x7fa7e00c7ff0 msgr2=0x7fa7e00c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:20.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.530+0000 7fa7e7476640 1 -- 172.21.15.107:0/1660367161 shutdown_connections 2023-12-04T05:47:20.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.530+0000 7fa7e7476640 1 -- 172.21.15.107:0/1660367161 wait complete. 2023-12-04T05:47:20.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.530+0000 7fa7e7476640 1 Processor -- start 2023-12-04T05:47:20.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.530+0000 7fa7e7476640 1 -- start start 2023-12-04T05:47:20.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.530+0000 7fa7e7476640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7cc004d10 0x7fa7e00ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:20.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.530+0000 7fa7e7476640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7e014d070 0x7fa7e006fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:20.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.530+0000 7fa7e7476640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa7e00718f0 0x7fa7e0148c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:20.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.530+0000 7fa7e7476640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e0150e00 con 0x7fa7e014d070 2023-12-04T05:47:20.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.530+0000 7fa7e7476640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e006e3c0 con 0x7fa7cc004d10 2023-12-04T05:47:20.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.530+0000 7fa7e7476640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e01eb560 con 0x7fa7e00718f0 2023-12-04T05:47:20.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.530+0000 7fa7e4a11640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7e014d070 0x7fa7e006fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:20.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.530+0000 7fa7e4a11640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7e014d070 0x7fa7e006fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41342/0 (socket says 172.21.15.107:41342) 2023-12-04T05:47:20.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.530+0000 7fa7e4a11640 1 -- 172.21.15.107:0/2348068044 learned_addr learned my addr 172.21.15.107:0/2348068044 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:20.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.534+0000 7fa7e5212640 1 --2- 172.21.15.107:0/2348068044 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7cc004d10 0x7fa7e00ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:20.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.534+0000 7fa7e4a11640 1 -- 172.21.15.107:0/2348068044 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa7e00718f0 msgr2=0x7fa7e0148c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:20.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.534+0000 7fa7e5a13640 1 --2- 172.21.15.107:0/2348068044 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa7e00718f0 0x7fa7e0148c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:20.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.534+0000 7fa7e4a11640 1 --2- 172.21.15.107:0/2348068044 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa7e00718f0 0x7fa7e0148c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:20.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.534+0000 7fa7e4a11640 1 -- 172.21.15.107:0/2348068044 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7cc004d10 msgr2=0x7fa7e00ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:20.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.534+0000 7fa7e4a11640 1 --2- 172.21.15.107:0/2348068044 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7cc004d10 0x7fa7e00ca5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:20.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.534+0000 7fa7e4a11640 1 -- 172.21.15.107:0/2348068044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e0070290 con 0x7fa7e014d070 2023-12-04T05:47:20.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.534+0000 7fa7e5212640 1 --2- 172.21.15.107:0/2348068044 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7cc004d10 0x7fa7e00ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:47:20.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.534+0000 7fa7e4a11640 1 --2- 172.21.15.107:0/2348068044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7e014d070 0x7fa7e006fd50 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fa7d00108d0 tx=0x7fa7d0010d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:20.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.534+0000 7fa7d5ffb640 1 -- 172.21.15.107:0/2348068044 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d0004220 con 0x7fa7e014d070 2023-12-04T05:47:20.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.534+0000 7fa7d5ffb640 1 -- 172.21.15.107:0/2348068044 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d0004380 con 0x7fa7e014d070 2023-12-04T05:47:20.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.534+0000 7fa7e7476640 1 -- 172.21.15.107:0/2348068044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e0150b20 con 0x7fa7e014d070 2023-12-04T05:47:20.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.534+0000 7fa7d5ffb640 1 -- 172.21.15.107:0/2348068044 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d000eba0 con 0x7fa7e014d070 2023-12-04T05:47:20.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.534+0000 7fa7e7476640 1 -- 172.21.15.107:0/2348068044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e006e660 con 0x7fa7e014d070 2023-12-04T05:47:20.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.534+0000 7fa7e7476640 1 -- 172.21.15.107:0/2348068044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e0067870 con 0x7fa7e014d070 2023-12-04T05:47:20.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.534+0000 7fa7d5ffb640 1 -- 172.21.15.107:0/2348068044 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa7d000ed00 con 0x7fa7e014d070 2023-12-04T05:47:20.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.534+0000 7fa7d5ffb640 1 --2- 172.21.15.107:0/2348068044 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa7c40668d0 0x7fa7c4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:20.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.534+0000 7fa7d5ffb640 1 -- 172.21.15.107:0/2348068044 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fa7d00844e0 con 0x7fa7e014d070 2023-12-04T05:47:20.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.534+0000 7fa7e5212640 1 --2- 172.21.15.107:0/2348068044 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa7c40668d0 0x7fa7c4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:20.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.534+0000 7fa7e5212640 1 --2- 172.21.15.107:0/2348068044 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa7c40668d0 0x7fa7c4068d90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc000c10 tx=0x7fa7dc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:20.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.538+0000 7fa7d5ffb640 1 -- 172.21.15.107:0/2348068044 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa7d00515e0 con 0x7fa7e014d070 2023-12-04T05:47:20.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.730+0000 7fa7e7476640 1 -- 172.21.15.107:0/2348068044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7e0158a00 con 0x7fa7e014d070 2023-12-04T05:47:20.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.730+0000 7fa7d5ffb640 1 -- 172.21.15.107:0/2348068044 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7fa7d0054c00 con 0x7fa7e014d070 2023-12-04T05:47:20.732 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:20.732 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:20.732 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:20.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.730+0000 7fa7e7476640 1 -- 172.21.15.107:0/2348068044 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa7c40668d0 msgr2=0x7fa7c4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:20.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.730+0000 7fa7e7476640 1 --2- 172.21.15.107:0/2348068044 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa7c40668d0 0x7fa7c4068d90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc000c10 tx=0x7fa7dc009040 comp rx=0 tx=0).stop 2023-12-04T05:47:20.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.730+0000 7fa7e7476640 1 -- 172.21.15.107:0/2348068044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7e014d070 msgr2=0x7fa7e006fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:20.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.730+0000 7fa7e7476640 1 --2- 172.21.15.107:0/2348068044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7e014d070 0x7fa7e006fd50 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fa7d00108d0 tx=0x7fa7d0010d90 comp rx=0 tx=0).stop 2023-12-04T05:47:20.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.730+0000 7fa7e7476640 1 -- 172.21.15.107:0/2348068044 shutdown_connections 2023-12-04T05:47:20.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.730+0000 7fa7e7476640 1 --2- 172.21.15.107:0/2348068044 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa7c40668d0 0x7fa7c4068d90 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:20.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.730+0000 7fa7e7476640 1 --2- 172.21.15.107:0/2348068044 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa7e00718f0 0x7fa7e0148c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:20.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.730+0000 7fa7e7476640 1 --2- 172.21.15.107:0/2348068044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7e014d070 0x7fa7e006fd50 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:20.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.730+0000 7fa7e7476640 1 --2- 172.21.15.107:0/2348068044 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7cc004d10 0x7fa7e00ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:20.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.730+0000 7fa7e7476640 1 -- 172.21.15.107:0/2348068044 >> 172.21.15.107:0/2348068044 conn(0x7fa7e00c7ff0 msgr2=0x7fa7e0156ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:20.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.730+0000 7fa7e7476640 1 -- 172.21.15.107:0/2348068044 shutdown_connections 2023-12-04T05:47:20.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:20.730+0000 7fa7e7476640 1 -- 172.21.15.107:0/2348068044 wait complete. 2023-12-04T05:47:20.751 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:47:20.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~99s 2023-12-04T05:47:20.949 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~213s 2023-12-04T05:47:20.949 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~254s 2023-12-04T05:47:22.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:22.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.878+0000 7fa4391d0640 1 Processor -- start 2023-12-04T05:47:22.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa4391d0640 1 -- start start 2023-12-04T05:47:22.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa4391d0640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa434071c00 0x7fa4340c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:22.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa4391d0640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fa434070380 0x7fa434070790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:22.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa4391d0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4340700c0 con 0x7fa434151180 2023-12-04T05:47:22.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa4391d0640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa434156df0 con 0x7fa434070380 2023-12-04T05:47:22.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa4391d0640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4341570d0 con 0x7fa434071c00 2023-12-04T05:47:22.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa432d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa434151180 0x7fa4340714f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54510/0 (socket says 172.21.15.107:54510) 2023-12-04T05:47:22.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa432d9d640 1 -- 172.21.15.107:0/2615438832 learned_addr learned my addr 172.21.15.107:0/2615438832 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:47:22.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa43259c640 1 --2- 172.21.15.107:0/2615438832 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa434071c00 0x7fa4340c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:22.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa43359e640 1 --2- 172.21.15.107:0/2615438832 >> v2:172.21.15.195:3300/0 conn(0x7fa434070380 0x7fa434070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:22.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa431d9b640 1 -- 172.21.15.107:0/2615438832 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380655140 0 0) 0x7fa4340700c0 con 0x7fa434151180 2023-12-04T05:47:22.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa431d9b640 1 -- 172.21.15.107:0/2615438832 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa41c003530 con 0x7fa434151180 2023-12-04T05:47:22.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa431d9b640 1 -- 172.21.15.107:0/2615438832 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2915361713 0 0) 0x7fa41c003530 con 0x7fa434151180 2023-12-04T05:47:22.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa431d9b640 1 -- 172.21.15.107:0/2615438832 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa434071c00 msgr2=0x7fa4340c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:22.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa431d9b640 1 --2- 172.21.15.107:0/2615438832 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa434071c00 0x7fa4340c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:22.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa431d9b640 1 -- 172.21.15.107:0/2615438832 >> v2:172.21.15.195:3300/0 conn(0x7fa434070380 msgr2=0x7fa434070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:22.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa431d9b640 1 --2- 172.21.15.107:0/2615438832 >> v2:172.21.15.195:3300/0 conn(0x7fa434070380 0x7fa434070790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:22.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa431d9b640 1 -- 172.21.15.107:0/2615438832 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4341573b0 con 0x7fa434151180 2023-12-04T05:47:22.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa431d9b640 1 -- 172.21.15.107:0/2615438832 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7fa424001680 con 0x7fa434151180 2023-12-04T05:47:22.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa431d9b640 1 -- 172.21.15.107:0/2615438832 >> v1:172.21.15.107:6789/0 conn(0x7fa434151180 legacy=0x7fa4340714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:22.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa431d9b640 1 --2- 172.21.15.107:0/2615438832 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa434071c00 0x7fa41c003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:22.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa431d9b640 1 --2- 172.21.15.107:0/2615438832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa41c003f00 0x7fa41c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:22.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa431d9b640 1 --2- 172.21.15.107:0/2615438832 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa41c004d10 0x7fa41c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:22.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa431d9b640 1 -- 172.21.15.107:0/2615438832 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa434156df0 con 0x7fa41c003f00 2023-12-04T05:47:22.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa431d9b640 1 -- 172.21.15.107:0/2615438832 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4341570d0 con 0x7fa41c004d10 2023-12-04T05:47:22.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa431d9b640 1 -- 172.21.15.107:0/2615438832 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c00b990 con 0x7fa434071c00 2023-12-04T05:47:22.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa43259c640 1 --2- 172.21.15.107:0/2615438832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa41c003f00 0x7fa41c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:22.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa43259c640 1 -- 172.21.15.107:0/2615438832 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa434071c00 msgr2=0x7fa41c003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:22.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa43259c640 1 --2- 172.21.15.107:0/2615438832 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa434071c00 0x7fa41c003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:22.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa43259c640 1 -- 172.21.15.107:0/2615438832 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa41c004d10 msgr2=0x7fa41c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:22.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa43259c640 1 --2- 172.21.15.107:0/2615438832 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa41c004d10 0x7fa41c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:22.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.882+0000 7fa43259c640 1 -- 172.21.15.107:0/2615438832 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41c003530 con 0x7fa41c003f00 2023-12-04T05:47:22.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa43259c640 1 --2- 172.21.15.107:0/2615438832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa41c003f00 0x7fa41c0047d0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fa428009ae0 tx=0x7fa42801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbaf73a788e8f2a0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:22.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa431d9b640 1 -- 172.21.15.107:0/2615438832 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4280a5680 con 0x7fa41c003f00 2023-12-04T05:47:22.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa431d9b640 1 -- 172.21.15.107:0/2615438832 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4280a57e0 con 0x7fa41c003f00 2023-12-04T05:47:22.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa431d9b640 1 -- 172.21.15.107:0/2615438832 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4280a5b90 con 0x7fa41c003f00 2023-12-04T05:47:22.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa4391d0640 1 -- 172.21.15.107:0/2615438832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa41c003f00 msgr2=0x7fa41c0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:22.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa4391d0640 1 --2- 172.21.15.107:0/2615438832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa41c003f00 0x7fa41c0047d0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fa428009ae0 tx=0x7fa42801ca30 comp rx=0 tx=0).stop 2023-12-04T05:47:22.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa43359e640 1 -- 172.21.15.107:0/2615438832 reap_dead start 2023-12-04T05:47:22.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa4391d0640 1 -- 172.21.15.107:0/2615438832 shutdown_connections 2023-12-04T05:47:22.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa4391d0640 1 -- 172.21.15.107:0/2615438832 >> 172.21.15.107:0/2615438832 conn(0x7fa4340c7ff0 msgr2=0x7fa4340c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:22.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa4391d0640 1 -- 172.21.15.107:0/2615438832 shutdown_connections 2023-12-04T05:47:22.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa4391d0640 1 -- 172.21.15.107:0/2615438832 wait complete. 2023-12-04T05:47:22.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa4391d0640 1 Processor -- start 2023-12-04T05:47:22.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa4391d0640 1 -- start start 2023-12-04T05:47:22.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa4391d0640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa41c003f00 0x7fa4340ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:22.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa4391d0640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4341490b0 0x7fa4341f3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:22.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa4391d0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa434149e80 0x7fa4341f3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:22.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa4391d0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4341f31b0 con 0x7fa434149e80 2023-12-04T05:47:22.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa4391d0640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43414aa50 con 0x7fa4341490b0 2023-12-04T05:47:22.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa4391d0640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa43414ad30 con 0x7fa41c003f00 2023-12-04T05:47:22.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa43359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa434149e80 0x7fa4341f3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:22.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa43359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa434149e80 0x7fa4341f3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41364/0 (socket says 172.21.15.107:41364) 2023-12-04T05:47:22.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa43359e640 1 -- 172.21.15.107:0/3819746697 learned_addr learned my addr 172.21.15.107:0/3819746697 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:22.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa432d9d640 1 --2- 172.21.15.107:0/3819746697 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa41c003f00 0x7fa4340ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:22.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa43259c640 1 --2- 172.21.15.107:0/3819746697 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4341490b0 0x7fa4341f3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:22.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa43359e640 1 -- 172.21.15.107:0/3819746697 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa41c003f00 msgr2=0x7fa4340ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:22.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa43359e640 1 --2- 172.21.15.107:0/3819746697 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa41c003f00 0x7fa4340ca7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:22.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa43359e640 1 -- 172.21.15.107:0/3819746697 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4341490b0 msgr2=0x7fa4341f3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:22.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa43359e640 1 --2- 172.21.15.107:0/3819746697 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4341490b0 0x7fa4341f3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:22.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa43359e640 1 -- 172.21.15.107:0/3819746697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa434146810 con 0x7fa434149e80 2023-12-04T05:47:22.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.886+0000 7fa43359e640 1 --2- 172.21.15.107:0/3819746697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa434149e80 0x7fa4341f3b40 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fa4340c7fc0 tx=0x7fa42c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:22.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.890+0000 7fa41b7fe640 1 -- 172.21.15.107:0/3819746697 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42c010040 con 0x7fa434149e80 2023-12-04T05:47:22.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.890+0000 7fa41b7fe640 1 -- 172.21.15.107:0/3819746697 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa42c00bae0 con 0x7fa434149e80 2023-12-04T05:47:22.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.890+0000 7fa4391d0640 1 -- 172.21.15.107:0/3819746697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa434146af0 con 0x7fa434149e80 2023-12-04T05:47:22.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.890+0000 7fa41b7fe640 1 -- 172.21.15.107:0/3819746697 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42c00be70 con 0x7fa434149e80 2023-12-04T05:47:22.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.890+0000 7fa4391d0640 1 -- 172.21.15.107:0/3819746697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa434156210 con 0x7fa434149e80 2023-12-04T05:47:22.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.890+0000 7fa4391d0640 1 -- 172.21.15.107:0/3819746697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa434067870 con 0x7fa434149e80 2023-12-04T05:47:22.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.890+0000 7fa41b7fe640 1 -- 172.21.15.107:0/3819746697 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fa42c01e410 con 0x7fa434149e80 2023-12-04T05:47:22.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.890+0000 7fa41b7fe640 1 --2- 172.21.15.107:0/3819746697 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa4200668d0 0x7fa420068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:22.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.890+0000 7fa41b7fe640 1 -- 172.21.15.107:0/3819746697 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fa42c084940 con 0x7fa434149e80 2023-12-04T05:47:22.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.890+0000 7fa432d9d640 1 --2- 172.21.15.107:0/3819746697 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa4200668d0 0x7fa420068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:22.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.890+0000 7fa432d9d640 1 --2- 172.21.15.107:0/3819746697 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa4200668d0 0x7fa420068d90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa424001030 tx=0x7fa424001670 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:22.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:22.894+0000 7fa41b7fe640 1 -- 172.21.15.107:0/3819746697 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa42c055440 con 0x7fa434149e80 2023-12-04T05:47:23.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:23.086+0000 7fa4391d0640 1 -- 172.21.15.107:0/3819746697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4340c26d0 con 0x7fa434149e80 2023-12-04T05:47:23.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:23.086+0000 7fa41b7fe640 1 -- 172.21.15.107:0/3819746697 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7fa42c054fd0 con 0x7fa434149e80 2023-12-04T05:47:23.088 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:23.088 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:23.088 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:23.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:23.086+0000 7fa4391d0640 1 -- 172.21.15.107:0/3819746697 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa4200668d0 msgr2=0x7fa420068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:23.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:23.086+0000 7fa4391d0640 1 --2- 172.21.15.107:0/3819746697 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa4200668d0 0x7fa420068d90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa424001030 tx=0x7fa424001670 comp rx=0 tx=0).stop 2023-12-04T05:47:23.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:23.086+0000 7fa4391d0640 1 -- 172.21.15.107:0/3819746697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa434149e80 msgr2=0x7fa4341f3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:23.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:23.086+0000 7fa4391d0640 1 --2- 172.21.15.107:0/3819746697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa434149e80 0x7fa4341f3b40 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fa4340c7fc0 tx=0x7fa42c00b480 comp rx=0 tx=0).stop 2023-12-04T05:47:23.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:23.086+0000 7fa4391d0640 1 -- 172.21.15.107:0/3819746697 shutdown_connections 2023-12-04T05:47:23.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:23.086+0000 7fa4391d0640 1 --2- 172.21.15.107:0/3819746697 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fa4200668d0 0x7fa420068d90 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:23.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:23.086+0000 7fa4391d0640 1 --2- 172.21.15.107:0/3819746697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa434149e80 0x7fa4341f3b40 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:23.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:23.086+0000 7fa4391d0640 1 --2- 172.21.15.107:0/3819746697 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa4341490b0 0x7fa4341f3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:23.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:23.086+0000 7fa4391d0640 1 --2- 172.21.15.107:0/3819746697 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fa41c003f00 0x7fa4340ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:23.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:23.086+0000 7fa4391d0640 1 -- 172.21.15.107:0/3819746697 >> 172.21.15.107:0/3819746697 conn(0x7fa4340c7ff0 msgr2=0x7fa434073960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:23.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:23.086+0000 7fa4391d0640 1 -- 172.21.15.107:0/3819746697 shutdown_connections 2023-12-04T05:47:23.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:23.086+0000 7fa4391d0640 1 -- 172.21.15.107:0/3819746697 wait complete. 2023-12-04T05:47:23.107 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:47:25.108 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:25.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.238+0000 7f846a4d9640 1 Processor -- start 2023-12-04T05:47:25.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.238+0000 7f846a4d9640 1 -- start start 2023-12-04T05:47:25.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.238+0000 7f846a4d9640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8464072110 0x7f8464158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:25.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.238+0000 7f846a4d9640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f84640c56f0 0x7f8464140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:25.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.238+0000 7f846a4d9640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8464158da0 con 0x7f84640724f0 2023-12-04T05:47:25.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.238+0000 7f846a4d9640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84641583d0 con 0x7f84640c56f0 2023-12-04T05:47:25.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.238+0000 7f846a4d9640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84640c6170 con 0x7f8464072110 2023-12-04T05:47:25.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.238+0000 7f8468a76640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f84640c56f0 0x7f8464140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:25.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.238+0000 7f8463fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8464072110 0x7f8464158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:25.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.238+0000 7f84637fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f84640724f0 0x7f84640c7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:57500/0 (socket says 172.21.15.107:57500) 2023-12-04T05:47:25.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.238+0000 7f84637fe640 1 -- 172.21.15.107:0/2169119777 learned_addr learned my addr 172.21.15.107:0/2169119777 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:47:25.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.238+0000 7f8468a76640 1 -- 172.21.15.107:0/2169119777 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8464072110 msgr2=0x7f8464158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:25.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.238+0000 7f8468a76640 1 --2- 172.21.15.107:0/2169119777 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8464072110 0x7f8464158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:25.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.238+0000 7f8468a76640 1 -- 172.21.15.107:0/2169119777 >> v1:172.21.15.195:6789/0 conn(0x7f84640724f0 legacy=0x7f84640c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:25.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.238+0000 7f8468a76640 1 -- 172.21.15.107:0/2169119777 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8464157da0 con 0x7f84640c56f0 2023-12-04T05:47:25.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.238+0000 7f8463fff640 1 --2- 172.21.15.107:0/2169119777 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8464072110 0x7f8464158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T05:47:25.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f8468a76640 1 --2- 172.21.15.107:0/2169119777 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84640c56f0 0x7f8464140830 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f845c0097c0 tx=0x7f845c01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f24eab4a4c841fc server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f8462ffd640 1 -- 172.21.15.107:0/2169119777 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845c0ab600 con 0x7f84640c56f0 2023-12-04T05:47:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f8462ffd640 1 -- 172.21.15.107:0/2169119777 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f845c0a7050 con 0x7f84640c56f0 2023-12-04T05:47:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f8462ffd640 1 -- 172.21.15.107:0/2169119777 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845c0ab980 con 0x7f84640c56f0 2023-12-04T05:47:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f846a4d9640 1 -- 172.21.15.107:0/2169119777 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84640c56f0 msgr2=0x7f8464140830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f846a4d9640 1 --2- 172.21.15.107:0/2169119777 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84640c56f0 0x7f8464140830 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f845c0097c0 tx=0x7f845c01c480 comp rx=0 tx=0).stop 2023-12-04T05:47:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f846a4d9640 1 -- 172.21.15.107:0/2169119777 shutdown_connections 2023-12-04T05:47:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f846a4d9640 1 --2- 172.21.15.107:0/2169119777 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84640c56f0 0x7f8464140830 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f846a4d9640 1 --2- 172.21.15.107:0/2169119777 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8464072110 0x7f8464158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f846a4d9640 1 -- 172.21.15.107:0/2169119777 >> 172.21.15.107:0/2169119777 conn(0x7f84640c7ff0 msgr2=0x7f84640c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f846a4d9640 1 -- 172.21.15.107:0/2169119777 shutdown_connections 2023-12-04T05:47:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f846a4d9640 1 -- 172.21.15.107:0/2169119777 wait complete. 2023-12-04T05:47:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f846a4d9640 1 Processor -- start 2023-12-04T05:47:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f846a4d9640 1 -- start start 2023-12-04T05:47:25.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f846a4d9640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8464072110 0x7f846413f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:25.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f846a4d9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84640724f0 0x7f846413fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f846a4d9640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84640c56f0 0x7f8464135210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f846a4d9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8464138790 con 0x7f84640724f0 2023-12-04T05:47:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f846a4d9640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84641374a0 con 0x7f84640c56f0 2023-12-04T05:47:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f846a4d9640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84641529e0 con 0x7f8464072110 2023-12-04T05:47:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f84637fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84640724f0 0x7f846413fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f84637fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84640724f0 0x7f846413fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41380/0 (socket says 172.21.15.107:41380) 2023-12-04T05:47:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.242+0000 7f84637fe640 1 -- 172.21.15.107:0/3808760045 learned_addr learned my addr 172.21.15.107:0/3808760045 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.246+0000 7f8463fff640 1 --2- 172.21.15.107:0/3808760045 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8464072110 0x7f846413f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.246+0000 7f8468a76640 1 --2- 172.21.15.107:0/3808760045 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84640c56f0 0x7f8464135210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.246+0000 7f84637fe640 1 -- 172.21.15.107:0/3808760045 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8464072110 msgr2=0x7f846413f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.246+0000 7f84637fe640 1 --2- 172.21.15.107:0/3808760045 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8464072110 0x7f846413f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.246+0000 7f84637fe640 1 -- 172.21.15.107:0/3808760045 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84640c56f0 msgr2=0x7f8464135210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.246+0000 7f84637fe640 1 --2- 172.21.15.107:0/3808760045 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84640c56f0 0x7f8464135210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.246+0000 7f84637fe640 1 -- 172.21.15.107:0/3808760045 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8454009420 con 0x7f84640724f0 2023-12-04T05:47:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.246+0000 7f84637fe640 1 --2- 172.21.15.107:0/3808760045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84640724f0 0x7f846413fce0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f844c00cbf0 tx=0x7f844c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.246+0000 7f8460ff9640 1 -- 172.21.15.107:0/3808760045 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844c00ad50 con 0x7f84640724f0 2023-12-04T05:47:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.246+0000 7f8460ff9640 1 -- 172.21.15.107:0/3808760045 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844c012400 con 0x7f84640724f0 2023-12-04T05:47:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.246+0000 7f8460ff9640 1 -- 172.21.15.107:0/3808760045 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844c012770 con 0x7f84640724f0 2023-12-04T05:47:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.246+0000 7f846a4d9640 1 -- 172.21.15.107:0/3808760045 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f845c009420 con 0x7f84640724f0 2023-12-04T05:47:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.246+0000 7f846a4d9640 1 -- 172.21.15.107:0/3808760045 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8464139ea0 con 0x7f84640724f0 2023-12-04T05:47:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.246+0000 7f846a4d9640 1 -- 172.21.15.107:0/3808760045 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8464067870 con 0x7f84640724f0 2023-12-04T05:47:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.246+0000 7f8460ff9640 1 -- 172.21.15.107:0/3808760045 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f844c007070 con 0x7f84640724f0 2023-12-04T05:47:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.246+0000 7f8460ff9640 1 --2- 172.21.15.107:0/3808760045 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f84440668d0 0x7f8444068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.246+0000 7f8460ff9640 1 -- 172.21.15.107:0/3808760045 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f844c0843a0 con 0x7f84640724f0 2023-12-04T05:47:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.246+0000 7f8463fff640 1 --2- 172.21.15.107:0/3808760045 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f84440668d0 0x7f8444068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.246+0000 7f8463fff640 1 --2- 172.21.15.107:0/3808760045 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f84440668d0 0x7f8444068d90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8454009c80 tx=0x7f8454008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:25.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.250+0000 7f8460ff9640 1 -- 172.21.15.107:0/3808760045 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f844c051430 con 0x7f84640724f0 2023-12-04T05:47:25.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.438+0000 7f846a4d9640 1 -- 172.21.15.107:0/3808760045 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8464158da0 con 0x7f84640724f0 2023-12-04T05:47:25.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.438+0000 7f8460ff9640 1 -- 172.21.15.107:0/3808760045 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f8464158da0 con 0x7f84640724f0 2023-12-04T05:47:25.441 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:25.442 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:25.442 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:25.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.442+0000 7f846a4d9640 1 -- 172.21.15.107:0/3808760045 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f84440668d0 msgr2=0x7f8444068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:25.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.442+0000 7f846a4d9640 1 --2- 172.21.15.107:0/3808760045 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f84440668d0 0x7f8444068d90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8454009c80 tx=0x7f8454008040 comp rx=0 tx=0).stop 2023-12-04T05:47:25.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.442+0000 7f846a4d9640 1 -- 172.21.15.107:0/3808760045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84640724f0 msgr2=0x7f846413fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:25.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.442+0000 7f846a4d9640 1 --2- 172.21.15.107:0/3808760045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84640724f0 0x7f846413fce0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f844c00cbf0 tx=0x7f844c00a430 comp rx=0 tx=0).stop 2023-12-04T05:47:25.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.442+0000 7f846a4d9640 1 -- 172.21.15.107:0/3808760045 shutdown_connections 2023-12-04T05:47:25.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.442+0000 7f846a4d9640 1 --2- 172.21.15.107:0/3808760045 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f84440668d0 0x7f8444068d90 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:25.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.442+0000 7f846a4d9640 1 --2- 172.21.15.107:0/3808760045 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84640c56f0 0x7f8464135210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:25.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.442+0000 7f846a4d9640 1 --2- 172.21.15.107:0/3808760045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84640724f0 0x7f846413fce0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:25.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.442+0000 7f846a4d9640 1 --2- 172.21.15.107:0/3808760045 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f8464072110 0x7f846413f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:25.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.442+0000 7f846a4d9640 1 -- 172.21.15.107:0/3808760045 >> 172.21.15.107:0/3808760045 conn(0x7f84640c7ff0 msgr2=0x7f84640c71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:25.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.442+0000 7f846a4d9640 1 -- 172.21.15.107:0/3808760045 shutdown_connections 2023-12-04T05:47:25.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:25.442+0000 7f846a4d9640 1 -- 172.21.15.107:0/3808760045 wait complete. 2023-12-04T05:47:25.460 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:47:27.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:27.561 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~106s 2023-12-04T05:47:27.561 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~220s 2023-12-04T05:47:27.561 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~261s 2023-12-04T05:47:27.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.590+0000 7fc9ac503640 1 Processor -- start 2023-12-04T05:47:27.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.590+0000 7fc9ac503640 1 -- start start 2023-12-04T05:47:27.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.594+0000 7fc9ac503640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc9a4151180 0x7fc9a40c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:27.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.594+0000 7fc9ac503640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fc9a4070380 0x7fc9a4070780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:27.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.594+0000 7fc9ac503640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a40700c0 con 0x7fc9a40714f0 2023-12-04T05:47:27.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.594+0000 7fc9ac503640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9a4156df0 con 0x7fc9a4151180 2023-12-04T05:47:27.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.594+0000 7fc9ac503640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9a41570d0 con 0x7fc9a4070380 2023-12-04T05:47:27.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.594+0000 7fc9aa29f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc9a4151180 0x7fc9a40c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:27.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.594+0000 7fc9aa29f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc9a4151180 0x7fc9a40c79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34482/0 (socket says 172.21.15.107:34482) 2023-12-04T05:47:27.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.594+0000 7fc9aa29f640 1 -- 172.21.15.107:0/983333991 learned_addr learned my addr 172.21.15.107:0/983333991 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:27.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.594+0000 7fc9aaaa0640 1 --2- 172.21.15.107:0/983333991 >> v2:172.21.15.195:3300/0 conn(0x7fc9a4070380 0x7fc9a4070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:27.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.594+0000 7fc9aa29f640 1 -- 172.21.15.107:0/983333991 >> v2:172.21.15.195:3300/0 conn(0x7fc9a4070380 msgr2=0x7fc9a4070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:27.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.594+0000 7fc9aa29f640 1 --2- 172.21.15.107:0/983333991 >> v2:172.21.15.195:3300/0 conn(0x7fc9a4070380 0x7fc9a4070780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:27.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.594+0000 7fc9aa29f640 1 -- 172.21.15.107:0/983333991 >> v1:172.21.15.195:6789/0 conn(0x7fc9a40714f0 legacy=0x7fc9a4071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:27.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.594+0000 7fc9aa29f640 1 -- 172.21.15.107:0/983333991 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a41573b0 con 0x7fc9a4151180 2023-12-04T05:47:27.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.594+0000 7fc9aa29f640 1 --2- 172.21.15.107:0/983333991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9a4151180 0x7fc9a40c79e0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc9a406f680 tx=0x7fc9a001ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c82f41520c3a8be server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:27.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.594+0000 7fc9a929d640 1 -- 172.21.15.107:0/983333991 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a00a89b0 con 0x7fc9a4151180 2023-12-04T05:47:27.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.594+0000 7fc9a929d640 1 -- 172.21.15.107:0/983333991 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a00a8b10 con 0x7fc9a4151180 2023-12-04T05:47:27.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.594+0000 7fc9a929d640 1 -- 172.21.15.107:0/983333991 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a00a8ea0 con 0x7fc9a4151180 2023-12-04T05:47:27.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9ac503640 1 -- 172.21.15.107:0/983333991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9a4151180 msgr2=0x7fc9a40c79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:27.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9ac503640 1 --2- 172.21.15.107:0/983333991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9a4151180 0x7fc9a40c79e0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc9a406f680 tx=0x7fc9a001ecb0 comp rx=0 tx=0).stop 2023-12-04T05:47:27.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9ac503640 1 -- 172.21.15.107:0/983333991 shutdown_connections 2023-12-04T05:47:27.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9ac503640 1 --2- 172.21.15.107:0/983333991 >> v2:172.21.15.195:3300/0 conn(0x7fc9a4070380 0x7fc9a4070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:27.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9ac503640 1 --2- 172.21.15.107:0/983333991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9a4151180 0x7fc9a40c79e0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:27.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9ac503640 1 -- 172.21.15.107:0/983333991 >> 172.21.15.107:0/983333991 conn(0x7fc9a40c7ff0 msgr2=0x7fc9a40c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:27.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9ac503640 1 -- 172.21.15.107:0/983333991 shutdown_connections 2023-12-04T05:47:27.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9ac503640 1 -- 172.21.15.107:0/983333991 wait complete. 2023-12-04T05:47:27.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9ac503640 1 Processor -- start 2023-12-04T05:47:27.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9ac503640 1 -- start start 2023-12-04T05:47:27.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9ac503640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc9a4070380 0x7fc9a4157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:27.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9ac503640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9a40714f0 0x7fc9a41e24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:27.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9ac503640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc9a4151180 0x7fc9a41449c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:27.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9ac503640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a415b0b0 con 0x7fc9a40714f0 2023-12-04T05:47:27.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9ac503640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a4159bc0 con 0x7fc9a4070380 2023-12-04T05:47:27.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9ac503640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a4158d10 con 0x7fc9a4151180 2023-12-04T05:47:27.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9a9a9e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9a40714f0 0x7fc9a41e24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:27.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9a9a9e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9a40714f0 0x7fc9a41e24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34492/0 (socket says 172.21.15.107:34492) 2023-12-04T05:47:27.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9a9a9e640 1 -- 172.21.15.107:0/1730110858 learned_addr learned my addr 172.21.15.107:0/1730110858 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:27.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9a9a9e640 1 -- 172.21.15.107:0/1730110858 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc9a4151180 msgr2=0x7fc9a41449c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:27.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9a9a9e640 1 --2- 172.21.15.107:0/1730110858 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc9a4151180 0x7fc9a41449c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:27.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9a9a9e640 1 -- 172.21.15.107:0/1730110858 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc9a4070380 msgr2=0x7fc9a4157e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:27.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9aa29f640 1 --2- 172.21.15.107:0/1730110858 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc9a4070380 0x7fc9a4157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:27.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9a9a9e640 1 --2- 172.21.15.107:0/1730110858 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc9a4070380 0x7fc9a4157e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:27.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9a9a9e640 1 -- 172.21.15.107:0/1730110858 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a00094d0 con 0x7fc9a40714f0 2023-12-04T05:47:27.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9a9a9e640 1 --2- 172.21.15.107:0/1730110858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9a40714f0 0x7fc9a41e24f0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fc99400c450 tx=0x7fc99400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:27.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc99affd640 1 -- 172.21.15.107:0/1730110858 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc994011790 con 0x7fc9a40714f0 2023-12-04T05:47:27.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc99affd640 1 -- 172.21.15.107:0/1730110858 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc994011910 con 0x7fc9a40714f0 2023-12-04T05:47:27.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9ac503640 1 -- 172.21.15.107:0/1730110858 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a4158fb0 con 0x7fc9a40714f0 2023-12-04T05:47:27.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc99affd640 1 -- 172.21.15.107:0/1730110858 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc994011cc0 con 0x7fc9a40714f0 2023-12-04T05:47:27.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.598+0000 7fc9ac503640 1 -- 172.21.15.107:0/1730110858 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a4159570 con 0x7fc9a40714f0 2023-12-04T05:47:27.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.602+0000 7fc9ac503640 1 -- 172.21.15.107:0/1730110858 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a4159d70 con 0x7fc9a40714f0 2023-12-04T05:47:27.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.602+0000 7fc99affd640 1 -- 172.21.15.107:0/1730110858 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fc99401c460 con 0x7fc9a40714f0 2023-12-04T05:47:27.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.602+0000 7fc99affd640 1 --2- 172.21.15.107:0/1730110858 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc9880668d0 0x7fc988068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:27.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.602+0000 7fc99affd640 1 -- 172.21.15.107:0/1730110858 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fc994084550 con 0x7fc9a40714f0 2023-12-04T05:47:27.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.602+0000 7fc9aa29f640 1 --2- 172.21.15.107:0/1730110858 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc9880668d0 0x7fc988068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:27.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.602+0000 7fc9aa29f640 1 --2- 172.21.15.107:0/1730110858 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc9880668d0 0x7fc988068d90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc9a00b2000 tx=0x7fc9a00aa070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:27.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.606+0000 7fc99affd640 1 -- 172.21.15.107:0/1730110858 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc9a4159d70 con 0x7fc9a40714f0 2023-12-04T05:47:27.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.794+0000 7fc9ac503640 1 -- 172.21.15.107:0/1730110858 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9a4159d70 con 0x7fc9a40714f0 2023-12-04T05:47:27.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.794+0000 7fc99affd640 1 -- 172.21.15.107:0/1730110858 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7fc9a4159d70 con 0x7fc9a40714f0 2023-12-04T05:47:27.796 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:27.796 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:27.796 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:27.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.794+0000 7fc9ac503640 1 -- 172.21.15.107:0/1730110858 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc9880668d0 msgr2=0x7fc988068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:27.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.794+0000 7fc9ac503640 1 --2- 172.21.15.107:0/1730110858 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc9880668d0 0x7fc988068d90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc9a00b2000 tx=0x7fc9a00aa070 comp rx=0 tx=0).stop 2023-12-04T05:47:27.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.794+0000 7fc9ac503640 1 -- 172.21.15.107:0/1730110858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9a40714f0 msgr2=0x7fc9a41e24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:27.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.794+0000 7fc9ac503640 1 --2- 172.21.15.107:0/1730110858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9a40714f0 0x7fc9a41e24f0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fc99400c450 tx=0x7fc99400c910 comp rx=0 tx=0).stop 2023-12-04T05:47:27.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.794+0000 7fc9ac503640 1 -- 172.21.15.107:0/1730110858 shutdown_connections 2023-12-04T05:47:27.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.794+0000 7fc9ac503640 1 --2- 172.21.15.107:0/1730110858 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fc9880668d0 0x7fc988068d90 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:27.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.794+0000 7fc9ac503640 1 --2- 172.21.15.107:0/1730110858 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fc9a4151180 0x7fc9a41449c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:27.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.794+0000 7fc9ac503640 1 --2- 172.21.15.107:0/1730110858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9a40714f0 0x7fc9a41e24f0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:27.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.794+0000 7fc9ac503640 1 --2- 172.21.15.107:0/1730110858 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc9a4070380 0x7fc9a4157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:27.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.794+0000 7fc9ac503640 1 -- 172.21.15.107:0/1730110858 >> 172.21.15.107:0/1730110858 conn(0x7fc9a40c7ff0 msgr2=0x7fc9a40c7510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:27.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.794+0000 7fc9ac503640 1 -- 172.21.15.107:0/1730110858 shutdown_connections 2023-12-04T05:47:27.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:27.794+0000 7fc9ac503640 1 -- 172.21.15.107:0/1730110858 wait complete. 2023-12-04T05:47:27.813 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 50, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-12-04T05:39:59.032019+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_4288": { "addr": "172.21.15.195:6837/3412865477", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6835", "nonce": 3412865477, "type": "v2" }, { "addr": "172.21.15.195:6837", "nonce": 3412865477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_5190": { "addr": "172.21.15.107:6836/2928623624", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 2928623624, "type": "v2" }, { "addr": "172.21.15.107:6836", "nonce": 2928623624, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5190, "incarnation": 45, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:rejoin", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-12-04T05:46:11.040247+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2, 3, 4 ], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 5190 } } } ], "standbys": [ { "addr": "172.21.15.107:6837/2752497349", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6835", "nonce": 2752497349, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 2752497349, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 49, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5292, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-12-04T05:47:29.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:29.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.938+0000 7f1e498db640 1 Processor -- start 2023-12-04T05:47:29.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.938+0000 7f1e498db640 1 -- start start 2023-12-04T05:47:29.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.942+0000 7f1e498db640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e44072110 0x7f1e44158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:29.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.942+0000 7f1e498db640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1e440c56f0 0x7f1e44140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:29.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.942+0000 7f1e498db640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e44158da0 con 0x7f1e440724f0 2023-12-04T05:47:29.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.942+0000 7f1e498db640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e441583d0 con 0x7f1e440c56f0 2023-12-04T05:47:29.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.942+0000 7f1e498db640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e440c6170 con 0x7f1e44072110 2023-12-04T05:47:29.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.942+0000 7f1e437fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1e440c56f0 0x7f1e44140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:29.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.942+0000 7f1e427fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1e440724f0 0x7f1e440c7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60072/0 (socket says 172.21.15.107:60072) 2023-12-04T05:47:29.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.942+0000 7f1e427fc640 1 -- 172.21.15.107:0/982450883 learned_addr learned my addr 172.21.15.107:0/982450883 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:47:29.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.942+0000 7f1e42ffd640 1 --2- 172.21.15.107:0/982450883 >> v2:172.21.15.195:3300/0 conn(0x7f1e44072110 0x7f1e44158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:29.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.942+0000 7f1e437fe640 1 -- 172.21.15.107:0/982450883 >> v2:172.21.15.195:3300/0 conn(0x7f1e44072110 msgr2=0x7f1e44158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:29.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.942+0000 7f1e437fe640 1 --2- 172.21.15.107:0/982450883 >> v2:172.21.15.195:3300/0 conn(0x7f1e44072110 0x7f1e44158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:29.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.942+0000 7f1e437fe640 1 -- 172.21.15.107:0/982450883 >> v1:172.21.15.107:6789/0 conn(0x7f1e440724f0 legacy=0x7f1e440c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:29.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.942+0000 7f1e437fe640 1 -- 172.21.15.107:0/982450883 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e44157da0 con 0x7f1e440c56f0 2023-12-04T05:47:29.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.942+0000 7f1e437fe640 1 --2- 172.21.15.107:0/982450883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e440c56f0 0x7f1e44140830 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f1e4406b8c0 tx=0x7f1e3c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5049d087fb0d3844 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:29.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.942+0000 7f1e41ffb640 1 -- 172.21.15.107:0/982450883 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c0a8950 con 0x7f1e440c56f0 2023-12-04T05:47:29.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.942+0000 7f1e41ffb640 1 -- 172.21.15.107:0/982450883 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3c0a8ab0 con 0x7f1e440c56f0 2023-12-04T05:47:29.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.942+0000 7f1e41ffb640 1 -- 172.21.15.107:0/982450883 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c0a8e40 con 0x7f1e440c56f0 2023-12-04T05:47:29.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e498db640 1 -- 172.21.15.107:0/982450883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e440c56f0 msgr2=0x7f1e44140830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:29.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e498db640 1 --2- 172.21.15.107:0/982450883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e440c56f0 0x7f1e44140830 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f1e4406b8c0 tx=0x7f1e3c01ec50 comp rx=0 tx=0).stop 2023-12-04T05:47:29.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e498db640 1 -- 172.21.15.107:0/982450883 shutdown_connections 2023-12-04T05:47:29.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e498db640 1 --2- 172.21.15.107:0/982450883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e440c56f0 0x7f1e44140830 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:29.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e498db640 1 --2- 172.21.15.107:0/982450883 >> v2:172.21.15.195:3300/0 conn(0x7f1e44072110 0x7f1e44158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:29.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e498db640 1 -- 172.21.15.107:0/982450883 >> 172.21.15.107:0/982450883 conn(0x7f1e440c7ff0 msgr2=0x7f1e440c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:29.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e498db640 1 -- 172.21.15.107:0/982450883 shutdown_connections 2023-12-04T05:47:29.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e498db640 1 -- 172.21.15.107:0/982450883 wait complete. 2023-12-04T05:47:29.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e498db640 1 Processor -- start 2023-12-04T05:47:29.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e498db640 1 -- start start 2023-12-04T05:47:29.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e498db640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e44072110 0x7f1e440ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:29.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e498db640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1e440724f0 0x7f1e4415b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:29.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e498db640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e440c56f0 0x7f1e4414cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:29.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e498db640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4415a8e0 con 0x7f1e440c56f0 2023-12-04T05:47:29.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e498db640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4415aed0 con 0x7f1e44072110 2023-12-04T05:47:29.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e498db640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4415c050 con 0x7f1e440724f0 2023-12-04T05:47:29.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e437fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e440c56f0 0x7f1e4414cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:29.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e437fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e440c56f0 0x7f1e4414cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34514/0 (socket says 172.21.15.107:34514) 2023-12-04T05:47:29.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e437fe640 1 -- 172.21.15.107:0/3604232078 learned_addr learned my addr 172.21.15.107:0/3604232078 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:29.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e42ffd640 1 --2- 172.21.15.107:0/3604232078 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e44072110 0x7f1e440ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:29.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e427fc640 1 --2- 172.21.15.107:0/3604232078 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1e440724f0 0x7f1e4415b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:29.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e437fe640 1 -- 172.21.15.107:0/3604232078 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1e440724f0 msgr2=0x7f1e4415b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:29.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e437fe640 1 --2- 172.21.15.107:0/3604232078 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1e440724f0 0x7f1e4415b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:29.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e437fe640 1 -- 172.21.15.107:0/3604232078 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e44072110 msgr2=0x7f1e440ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:29.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e437fe640 1 --2- 172.21.15.107:0/3604232078 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e44072110 0x7f1e440ca6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:29.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e437fe640 1 -- 172.21.15.107:0/3604232078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e3c009420 con 0x7f1e440c56f0 2023-12-04T05:47:29.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e427fc640 1 --2- 172.21.15.107:0/3604232078 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1e440724f0 0x7f1e4415b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:47:29.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e437fe640 1 --2- 172.21.15.107:0/3604232078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e440c56f0 0x7f1e4414cf70 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c0b3040 tx=0x7f1e3c01b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:29.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e3bfff640 1 -- 172.21.15.107:0/3604232078 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c0b9630 con 0x7f1e440c56f0 2023-12-04T05:47:29.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e3bfff640 1 -- 172.21.15.107:0/3604232078 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3c0b9790 con 0x7f1e440c56f0 2023-12-04T05:47:29.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e498db640 1 -- 172.21.15.107:0/3604232078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e441556a0 con 0x7f1e440c56f0 2023-12-04T05:47:29.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e3bfff640 1 -- 172.21.15.107:0/3604232078 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c0b9b40 con 0x7f1e440c56f0 2023-12-04T05:47:29.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.946+0000 7f1e498db640 1 -- 172.21.15.107:0/3604232078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e44154e00 con 0x7f1e440c56f0 2023-12-04T05:47:29.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.950+0000 7f1e498db640 1 -- 172.21.15.107:0/3604232078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e441f29d0 con 0x7f1e440c56f0 2023-12-04T05:47:29.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.950+0000 7f1e3bfff640 1 -- 172.21.15.107:0/3604232078 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f1e3c0b7040 con 0x7f1e440c56f0 2023-12-04T05:47:29.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.950+0000 7f1e3bfff640 1 --2- 172.21.15.107:0/3604232078 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1e24066900 0x7f1e24068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:29.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.950+0000 7f1e42ffd640 1 --2- 172.21.15.107:0/3604232078 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1e24066900 0x7f1e24068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:29.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.950+0000 7f1e3bfff640 1 -- 172.21.15.107:0/3604232078 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f1e3c1224a0 con 0x7f1e440c56f0 2023-12-04T05:47:29.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.950+0000 7f1e42ffd640 1 --2- 172.21.15.107:0/3604232078 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1e24066900 0x7f1e24068dc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1e3400b9d0 tx=0x7f1e34008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:29.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:29.954+0000 7f1e3bfff640 1 -- 172.21.15.107:0/3604232078 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1e441f29d0 con 0x7f1e440c56f0 2023-12-04T05:47:30.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:30.130+0000 7f1e498db640 1 -- 172.21.15.107:0/3604232078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e441f29d0 con 0x7f1e440c56f0 2023-12-04T05:47:30.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:30.130+0000 7f1e3bfff640 1 -- 172.21.15.107:0/3604232078 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f1e441f29d0 con 0x7f1e440c56f0 2023-12-04T05:47:30.134 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:30.134 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:30.134 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:30.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:30.134+0000 7f1e498db640 1 -- 172.21.15.107:0/3604232078 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1e24066900 msgr2=0x7f1e24068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:30.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:30.134+0000 7f1e498db640 1 --2- 172.21.15.107:0/3604232078 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1e24066900 0x7f1e24068dc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1e3400b9d0 tx=0x7f1e34008040 comp rx=0 tx=0).stop 2023-12-04T05:47:30.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:30.134+0000 7f1e498db640 1 -- 172.21.15.107:0/3604232078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e440c56f0 msgr2=0x7f1e4414cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:30.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:30.134+0000 7f1e498db640 1 --2- 172.21.15.107:0/3604232078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e440c56f0 0x7f1e4414cf70 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c0b3040 tx=0x7f1e3c01b9f0 comp rx=0 tx=0).stop 2023-12-04T05:47:30.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:30.134+0000 7f1e498db640 1 -- 172.21.15.107:0/3604232078 shutdown_connections 2023-12-04T05:47:30.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:30.134+0000 7f1e498db640 1 --2- 172.21.15.107:0/3604232078 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1e24066900 0x7f1e24068dc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:30.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:30.134+0000 7f1e498db640 1 --2- 172.21.15.107:0/3604232078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e440c56f0 0x7f1e4414cf70 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:30.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:30.134+0000 7f1e498db640 1 --2- 172.21.15.107:0/3604232078 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1e440724f0 0x7f1e4415b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:30.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:30.134+0000 7f1e498db640 1 --2- 172.21.15.107:0/3604232078 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e44072110 0x7f1e440ca6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:30.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:30.134+0000 7f1e498db640 1 -- 172.21.15.107:0/3604232078 >> 172.21.15.107:0/3604232078 conn(0x7f1e440c7ff0 msgr2=0x7f1e44155c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:30.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:30.134+0000 7f1e498db640 1 -- 172.21.15.107:0/3604232078 shutdown_connections 2023-12-04T05:47:30.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:30.134+0000 7f1e498db640 1 -- 172.21.15.107:0/3604232078 wait complete. 2023-12-04T05:47:30.152 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:47:31.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:47:31.196 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:47:32.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:32.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.286+0000 7f62e8083640 1 Processor -- start 2023-12-04T05:47:32.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.286+0000 7f62e8083640 1 -- start start 2023-12-04T05:47:32.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.286+0000 7f62e8083640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f62e0151180 0x7f62e00c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:32.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.286+0000 7f62e8083640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f62e0070380 0x7f62e00707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:32.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.286+0000 7f62e8083640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e00700c0 con 0x7f62e00714f0 2023-12-04T05:47:32.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.286+0000 7f62e8083640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62e0156df0 con 0x7f62e0151180 2023-12-04T05:47:32.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.286+0000 7f62e8083640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62e01570d0 con 0x7f62e0070380 2023-12-04T05:47:32.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.286+0000 7f62e5e1f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f62e0151180 0x7f62e00c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:32.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.286+0000 7f62e561e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f62e00714f0 0x7f62e0071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60080/0 (socket says 172.21.15.107:60080) 2023-12-04T05:47:32.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.286+0000 7f62e561e640 1 -- 172.21.15.107:0/3787661800 learned_addr learned my addr 172.21.15.107:0/3787661800 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:47:32.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.286+0000 7f62e5e1f640 1 -- 172.21.15.107:0/3787661800 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f62e0070380 msgr2=0x7f62e00707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:32.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.286+0000 7f62e5e1f640 1 --2- 172.21.15.107:0/3787661800 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f62e0070380 0x7f62e00707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:32.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.286+0000 7f62e5e1f640 1 -- 172.21.15.107:0/3787661800 >> v1:172.21.15.107:6789/0 conn(0x7f62e00714f0 legacy=0x7f62e0071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:32.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.286+0000 7f62e5e1f640 1 -- 172.21.15.107:0/3787661800 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e01573b0 con 0x7f62e0151180 2023-12-04T05:47:32.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e5e1f640 1 --2- 172.21.15.107:0/3787661800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62e0151180 0x7f62e00c79e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0097c0 tx=0x7f62dc01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3db1f7d4e7446527 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:32.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e4e1d640 1 -- 172.21.15.107:0/3787661800 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62dc0a8970 con 0x7f62e0151180 2023-12-04T05:47:32.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e4e1d640 1 -- 172.21.15.107:0/3787661800 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62dc0a8ad0 con 0x7f62e0151180 2023-12-04T05:47:32.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e4e1d640 1 -- 172.21.15.107:0/3787661800 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62dc0a8e60 con 0x7f62e0151180 2023-12-04T05:47:32.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e8083640 1 -- 172.21.15.107:0/3787661800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62e0151180 msgr2=0x7f62e00c79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:32.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e8083640 1 --2- 172.21.15.107:0/3787661800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62e0151180 0x7f62e00c79e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0097c0 tx=0x7f62dc01ec70 comp rx=0 tx=0).stop 2023-12-04T05:47:32.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e8083640 1 -- 172.21.15.107:0/3787661800 shutdown_connections 2023-12-04T05:47:32.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e8083640 1 --2- 172.21.15.107:0/3787661800 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f62e0070380 0x7f62e00707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:32.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e8083640 1 --2- 172.21.15.107:0/3787661800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62e0151180 0x7f62e00c79e0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:32.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e8083640 1 -- 172.21.15.107:0/3787661800 >> 172.21.15.107:0/3787661800 conn(0x7f62e00c7ff0 msgr2=0x7f62e00c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:32.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e8083640 1 -- 172.21.15.107:0/3787661800 shutdown_connections 2023-12-04T05:47:32.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e8083640 1 -- 172.21.15.107:0/3787661800 wait complete. 2023-12-04T05:47:32.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e8083640 1 Processor -- start 2023-12-04T05:47:32.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e8083640 1 -- start start 2023-12-04T05:47:32.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e8083640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f62e0070380 0x7f62e013c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:32.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e8083640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f62e00714f0 0x7f62e013c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:32.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e8083640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62e0151180 0x7f62e01468b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:32.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e8083640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e0145e80 con 0x7f62e0151180 2023-12-04T05:47:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e8083640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e00c4b10 con 0x7f62e00714f0 2023-12-04T05:47:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e8083640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62e01460d0 con 0x7f62e0070380 2023-12-04T05:47:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e6620640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62e0151180 0x7f62e01468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e6620640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62e0151180 0x7f62e01468b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34526/0 (socket says 172.21.15.107:34526) 2023-12-04T05:47:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e6620640 1 -- 172.21.15.107:0/1524663842 learned_addr learned my addr 172.21.15.107:0/1524663842 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e5e1f640 1 --2- 172.21.15.107:0/1524663842 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f62e0070380 0x7f62e013c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e6620640 1 -- 172.21.15.107:0/1524663842 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f62e0070380 msgr2=0x7f62e013c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e6620640 1 --2- 172.21.15.107:0/1524663842 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f62e0070380 0x7f62e013c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e6620640 1 -- 172.21.15.107:0/1524663842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f62e00714f0 msgr2=0x7f62e013c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e6620640 1 --2- 172.21.15.107:0/1524663842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f62e00714f0 0x7f62e013c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e6620640 1 -- 172.21.15.107:0/1524663842 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62dc009420 con 0x7f62e0151180 2023-12-04T05:47:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e6620640 1 --2- 172.21.15.107:0/1524663842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62e0151180 0x7f62e01468b0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f62d800b970 tx=0x7f62d800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:32.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62d67fc640 1 -- 172.21.15.107:0/1524663842 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d800cb40 con 0x7f62e0151180 2023-12-04T05:47:32.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62d67fc640 1 -- 172.21.15.107:0/1524663842 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d800ccc0 con 0x7f62e0151180 2023-12-04T05:47:32.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e8083640 1 -- 172.21.15.107:0/1524663842 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e00cac50 con 0x7f62e0151180 2023-12-04T05:47:32.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62d67fc640 1 -- 172.21.15.107:0/1524663842 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d8014680 con 0x7f62e0151180 2023-12-04T05:47:32.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.290+0000 7f62e8083640 1 -- 172.21.15.107:0/1524663842 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e01463b0 con 0x7f62e0151180 2023-12-04T05:47:32.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.294+0000 7f62e8083640 1 -- 172.21.15.107:0/1524663842 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e00cb170 con 0x7f62e0151180 2023-12-04T05:47:32.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.294+0000 7f62d67fc640 1 -- 172.21.15.107:0/1524663842 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f62d801d3b0 con 0x7f62e0151180 2023-12-04T05:47:32.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.294+0000 7f62d67fc640 1 --2- 172.21.15.107:0/1524663842 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f62c40668d0 0x7f62c4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:32.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.294+0000 7f62d67fc640 1 -- 172.21.15.107:0/1524663842 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f62d8084600 con 0x7f62e0151180 2023-12-04T05:47:32.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.294+0000 7f62e5e1f640 1 --2- 172.21.15.107:0/1524663842 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f62c40668d0 0x7f62c4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:32.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.294+0000 7f62e5e1f640 1 --2- 172.21.15.107:0/1524663842 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f62c40668d0 0x7f62c4068d90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0b2000 tx=0x7f62dc0aa070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:32.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.298+0000 7f62d67fc640 1 -- 172.21.15.107:0/1524663842 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f62e00cb170 con 0x7f62e0151180 2023-12-04T05:47:32.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.486+0000 7f62e8083640 1 -- 172.21.15.107:0/1524663842 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62e00cb170 con 0x7f62e0151180 2023-12-04T05:47:32.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.486+0000 7f62d67fc640 1 -- 172.21.15.107:0/1524663842 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f62e00cb170 con 0x7f62e0151180 2023-12-04T05:47:32.488 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:32.488 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:32.488 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:32.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.486+0000 7f62e8083640 1 -- 172.21.15.107:0/1524663842 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f62c40668d0 msgr2=0x7f62c4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:32.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.486+0000 7f62e8083640 1 --2- 172.21.15.107:0/1524663842 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f62c40668d0 0x7f62c4068d90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0b2000 tx=0x7f62dc0aa070 comp rx=0 tx=0).stop 2023-12-04T05:47:32.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.486+0000 7f62e8083640 1 -- 172.21.15.107:0/1524663842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62e0151180 msgr2=0x7f62e01468b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:32.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.486+0000 7f62e8083640 1 --2- 172.21.15.107:0/1524663842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62e0151180 0x7f62e01468b0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f62d800b970 tx=0x7f62d800be30 comp rx=0 tx=0).stop 2023-12-04T05:47:32.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.486+0000 7f62e8083640 1 -- 172.21.15.107:0/1524663842 shutdown_connections 2023-12-04T05:47:32.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.486+0000 7f62e8083640 1 --2- 172.21.15.107:0/1524663842 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f62c40668d0 0x7f62c4068d90 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:32.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.486+0000 7f62e8083640 1 --2- 172.21.15.107:0/1524663842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62e0151180 0x7f62e01468b0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:32.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.486+0000 7f62e8083640 1 --2- 172.21.15.107:0/1524663842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f62e00714f0 0x7f62e013c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:32.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.486+0000 7f62e8083640 1 --2- 172.21.15.107:0/1524663842 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f62e0070380 0x7f62e013c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:32.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.486+0000 7f62e8083640 1 -- 172.21.15.107:0/1524663842 >> 172.21.15.107:0/1524663842 conn(0x7f62e00c7ff0 msgr2=0x7f62e00c7540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:32.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.490+0000 7f62e8083640 1 -- 172.21.15.107:0/1524663842 shutdown_connections 2023-12-04T05:47:32.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:32.490+0000 7f62e8083640 1 -- 172.21.15.107:0/1524663842 wait complete. 2023-12-04T05:47:32.507 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:47:34.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~112s 2023-12-04T05:47:34.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~227s 2023-12-04T05:47:34.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~267s 2023-12-04T05:47:34.508 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:34.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.634+0000 7f180a529640 1 Processor -- start 2023-12-04T05:47:34.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.634+0000 7f180a529640 1 -- start start 2023-12-04T05:47:34.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f180a529640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f18040c8c80 0x7f180406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:34.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f180a529640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f180406e320 con 0x7f18040c6b70 2023-12-04T05:47:34.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f180a529640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18040c66e0 con 0x7f1804154290 2023-12-04T05:47:34.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f180a529640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1804071c10 con 0x7f18040c8c80 2023-12-04T05:47:34.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1808ac6640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f18040c6b70 0x7f180415a710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60094/0 (socket says 172.21.15.107:60094) 2023-12-04T05:47:34.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1808ac6640 1 -- 172.21.15.107:0/396605715 learned_addr learned my addr 172.21.15.107:0/396605715 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:47:34.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f18037fe640 1 --2- 172.21.15.107:0/396605715 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f18040c8c80 0x7f180406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:34.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1802ffd640 1 -- 172.21.15.107:0/396605715 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786799640 0 0) 0x7f180406e320 con 0x7f18040c6b70 2023-12-04T05:47:34.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1802ffd640 1 -- 172.21.15.107:0/396605715 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17ec003530 con 0x7f18040c6b70 2023-12-04T05:47:34.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1802ffd640 1 -- 172.21.15.107:0/396605715 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 486587911 0 0) 0x7f17ec003530 con 0x7f18040c6b70 2023-12-04T05:47:34.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1802ffd640 1 -- 172.21.15.107:0/396605715 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f18040c8c80 msgr2=0x7f180406fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:34.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1802ffd640 1 --2- 172.21.15.107:0/396605715 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f18040c8c80 0x7f180406fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:34.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1802ffd640 1 -- 172.21.15.107:0/396605715 >> v1:172.21.15.195:6789/0 conn(0x7f1804154290 legacy=0x7f180415a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:34.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1802ffd640 1 -- 172.21.15.107:0/396605715 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180415ae20 con 0x7f18040c6b70 2023-12-04T05:47:34.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1802ffd640 1 -- 172.21.15.107:0/396605715 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f17fc001a00 con 0x7f18040c6b70 2023-12-04T05:47:34.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1802ffd640 1 -- 172.21.15.107:0/396605715 >> v1:172.21.15.107:6789/0 conn(0x7f18040c6b70 legacy=0x7f180415a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:34.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1802ffd640 1 --2- 172.21.15.107:0/396605715 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f18040c8c80 0x7f17ec0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:34.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1802ffd640 1 --2- 172.21.15.107:0/396605715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f17ec003f30 0x7f17ec0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:34.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1802ffd640 1 --2- 172.21.15.107:0/396605715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17ec004d10 0x7f17ec005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:34.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1802ffd640 1 -- 172.21.15.107:0/396605715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec00bba0 con 0x7f17ec004d10 2023-12-04T05:47:34.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1802ffd640 1 -- 172.21.15.107:0/396605715 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec00bd10 con 0x7f17ec003f30 2023-12-04T05:47:34.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1802ffd640 1 -- 172.21.15.107:0/396605715 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec00bed0 con 0x7f18040c8c80 2023-12-04T05:47:34.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1808ac6640 1 --2- 172.21.15.107:0/396605715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17ec004d10 0x7f17ec005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:34.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1803fff640 1 --2- 172.21.15.107:0/396605715 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f18040c8c80 0x7f17ec0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:34.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f18037fe640 1 --2- 172.21.15.107:0/396605715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f17ec003f30 0x7f17ec0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:34.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1808ac6640 1 -- 172.21.15.107:0/396605715 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f18040c8c80 msgr2=0x7f17ec0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:34.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1808ac6640 1 --2- 172.21.15.107:0/396605715 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f18040c8c80 0x7f17ec0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:34.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1808ac6640 1 -- 172.21.15.107:0/396605715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f17ec003f30 msgr2=0x7f17ec0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:34.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1808ac6640 1 --2- 172.21.15.107:0/396605715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f17ec003f30 0x7f17ec0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:34.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.638+0000 7f1808ac6640 1 -- 172.21.15.107:0/396605715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ec003530 con 0x7f17ec004d10 2023-12-04T05:47:34.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f1808ac6640 1 --2- 172.21.15.107:0/396605715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17ec004d10 0x7f17ec005190 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f17fc00a830 tx=0x7f17fc01d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b16cadbce654b2d1 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:34.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f1802ffd640 1 -- 172.21.15.107:0/396605715 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17fc0072d0 con 0x7f17ec004d10 2023-12-04T05:47:34.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f1802ffd640 1 -- 172.21.15.107:0/396605715 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17fc0a8070 con 0x7f17ec004d10 2023-12-04T05:47:34.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f1802ffd640 1 -- 172.21.15.107:0/396605715 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17fc0076f0 con 0x7f17ec004d10 2023-12-04T05:47:34.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f180a529640 1 -- 172.21.15.107:0/396605715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17ec004d10 msgr2=0x7f17ec005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:34.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f180a529640 1 --2- 172.21.15.107:0/396605715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17ec004d10 0x7f17ec005190 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f17fc00a830 tx=0x7f17fc01d340 comp rx=0 tx=0).stop 2023-12-04T05:47:34.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f1808ac6640 1 -- 172.21.15.107:0/396605715 reap_dead start 2023-12-04T05:47:34.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f180a529640 1 -- 172.21.15.107:0/396605715 shutdown_connections 2023-12-04T05:47:34.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f180a529640 1 -- 172.21.15.107:0/396605715 >> 172.21.15.107:0/396605715 conn(0x7f18040c7ff0 msgr2=0x7f18040c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:34.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f180a529640 1 -- 172.21.15.107:0/396605715 shutdown_connections 2023-12-04T05:47:34.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f180a529640 1 -- 172.21.15.107:0/396605715 wait complete. 2023-12-04T05:47:34.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f180a529640 1 Processor -- start 2023-12-04T05:47:34.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f180a529640 1 -- start start 2023-12-04T05:47:34.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f180a529640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17ec004d10 0x7f18040ca520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:34.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f180a529640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f18041f68b0 0x7f18041f26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:34.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f180a529640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18041559f0 0x7f180414a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:34.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f180a529640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18040caa60 con 0x7f17ec004d10 2023-12-04T05:47:34.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f180a529640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1804154e30 con 0x7f18041559f0 2023-12-04T05:47:34.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f180a529640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18040cac60 con 0x7f18041f68b0 2023-12-04T05:47:34.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f1803fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17ec004d10 0x7f18040ca520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:34.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f1803fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17ec004d10 0x7f18040ca520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34534/0 (socket says 172.21.15.107:34534) 2023-12-04T05:47:34.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f1803fff640 1 -- 172.21.15.107:0/668643095 learned_addr learned my addr 172.21.15.107:0/668643095 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:34.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f18037fe640 1 --2- 172.21.15.107:0/668643095 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f18041f68b0 0x7f18041f26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:34.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f1808ac6640 1 --2- 172.21.15.107:0/668643095 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18041559f0 0x7f180414a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:34.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f1803fff640 1 -- 172.21.15.107:0/668643095 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f18041f68b0 msgr2=0x7f18041f26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:34.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f1803fff640 1 --2- 172.21.15.107:0/668643095 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f18041f68b0 0x7f18041f26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:34.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f1803fff640 1 -- 172.21.15.107:0/668643095 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18041559f0 msgr2=0x7f180414a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:34.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f1803fff640 1 --2- 172.21.15.107:0/668643095 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18041559f0 0x7f180414a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:34.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f1803fff640 1 -- 172.21.15.107:0/668643095 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18040caf40 con 0x7f17ec004d10 2023-12-04T05:47:34.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f1803fff640 1 --2- 172.21.15.107:0/668643095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17ec004d10 0x7f18040ca520 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f17f400e550 tx=0x7f17f400ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:34.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.642+0000 7f1800ff9640 1 -- 172.21.15.107:0/668643095 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f40052e0 con 0x7f17ec004d10 2023-12-04T05:47:34.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.646+0000 7f1800ff9640 1 -- 172.21.15.107:0/668643095 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f4005440 con 0x7f17ec004d10 2023-12-04T05:47:34.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.646+0000 7f180a529640 1 -- 172.21.15.107:0/668643095 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18041f5dd0 con 0x7f17ec004d10 2023-12-04T05:47:34.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.646+0000 7f1800ff9640 1 -- 172.21.15.107:0/668643095 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f4003820 con 0x7f17ec004d10 2023-12-04T05:47:34.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.646+0000 7f180a529640 1 -- 172.21.15.107:0/668643095 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18041f6390 con 0x7f17ec004d10 2023-12-04T05:47:34.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.646+0000 7f180a529640 1 -- 172.21.15.107:0/668643095 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18041f5c30 con 0x7f17ec004d10 2023-12-04T05:47:34.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.646+0000 7f1800ff9640 1 -- 172.21.15.107:0/668643095 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f17f4003980 con 0x7f17ec004d10 2023-12-04T05:47:34.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.646+0000 7f1800ff9640 1 --2- 172.21.15.107:0/668643095 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f17f0066900 0x7f17f0068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:34.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.646+0000 7f1800ff9640 1 -- 172.21.15.107:0/668643095 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f17f4083f70 con 0x7f17ec004d10 2023-12-04T05:47:34.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.646+0000 7f18037fe640 1 --2- 172.21.15.107:0/668643095 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f17f0066900 0x7f17f0068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:34.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.646+0000 7f18037fe640 1 --2- 172.21.15.107:0/668643095 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f17f0066900 0x7f17f0068dc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1804165740 tx=0x7f17f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:34.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.650+0000 7f1800ff9640 1 -- 172.21.15.107:0/668643095 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f17f40b5280 con 0x7f17ec004d10 2023-12-04T05:47:34.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.830+0000 7f180a529640 1 -- 172.21.15.107:0/668643095 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18041f6520 con 0x7f17ec004d10 2023-12-04T05:47:34.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.830+0000 7f1800ff9640 1 -- 172.21.15.107:0/668643095 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f18041f5c30 con 0x7f17ec004d10 2023-12-04T05:47:34.834 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:34.834 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:34.834 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.834+0000 7f180a529640 1 -- 172.21.15.107:0/668643095 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f17f0066900 msgr2=0x7f17f0068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.834+0000 7f180a529640 1 --2- 172.21.15.107:0/668643095 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f17f0066900 0x7f17f0068dc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1804165740 tx=0x7f17f8008040 comp rx=0 tx=0).stop 2023-12-04T05:47:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.834+0000 7f180a529640 1 -- 172.21.15.107:0/668643095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17ec004d10 msgr2=0x7f18040ca520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.834+0000 7f180a529640 1 --2- 172.21.15.107:0/668643095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17ec004d10 0x7f18040ca520 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f17f400e550 tx=0x7f17f400ea10 comp rx=0 tx=0).stop 2023-12-04T05:47:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.834+0000 7f180a529640 1 -- 172.21.15.107:0/668643095 shutdown_connections 2023-12-04T05:47:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.834+0000 7f180a529640 1 --2- 172.21.15.107:0/668643095 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f17f0066900 0x7f17f0068dc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.834+0000 7f180a529640 1 --2- 172.21.15.107:0/668643095 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18041559f0 0x7f180414a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.834+0000 7f180a529640 1 --2- 172.21.15.107:0/668643095 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f18041f68b0 0x7f18041f26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.834+0000 7f180a529640 1 --2- 172.21.15.107:0/668643095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17ec004d10 0x7f18040ca520 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.834+0000 7f180a529640 1 -- 172.21.15.107:0/668643095 >> 172.21.15.107:0/668643095 conn(0x7f18040c7ff0 msgr2=0x7f18040719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:34.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.834+0000 7f180a529640 1 -- 172.21.15.107:0/668643095 shutdown_connections 2023-12-04T05:47:34.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:34.834+0000 7f180a529640 1 -- 172.21.15.107:0/668643095 wait complete. 2023-12-04T05:47:34.853 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:47:36.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:36.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.982+0000 7fb6c3948640 1 Processor -- start 2023-12-04T05:47:36.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.982+0000 7fb6c3948640 1 -- start start 2023-12-04T05:47:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.986+0000 7fb6c3948640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb6bc154020 0x7fb6bc15a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.986+0000 7fb6c3948640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6bc0d01c0 0x7fb6bc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.986+0000 7fb6c3948640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb6bc072220 0x7fb6bc0c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.986+0000 7fb6c3948640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6bc0ca230 con 0x7fb6bc154020 2023-12-04T05:47:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.986+0000 7fb6c3948640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6bc0c6770 con 0x7fb6bc072220 2023-12-04T05:47:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.986+0000 7fb6c3948640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc15ad50 con 0x7fb6bc0d01c0 2023-12-04T05:47:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.986+0000 7fb6c16e4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb6bc154020 0x7fb6bc15a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.986+0000 7fb6c16e4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb6bc154020 0x7fb6bc15a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34536/0 (socket says 172.21.15.107:34536) 2023-12-04T05:47:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.986+0000 7fb6c16e4640 1 -- 172.21.15.107:0/974783638 learned_addr learned my addr 172.21.15.107:0/974783638 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.986+0000 7fb6c0ee3640 1 --2- 172.21.15.107:0/974783638 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6bc0d01c0 0x7fb6bc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:36.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.986+0000 7fb6c1ee5640 1 --2- 172.21.15.107:0/974783638 >> v2:172.21.15.195:3300/0 conn(0x7fb6bc072220 0x7fb6bc0c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:36.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.986+0000 7fb6c16e4640 1 -- 172.21.15.107:0/974783638 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6bc0d01c0 msgr2=0x7fb6bc071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:36.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.986+0000 7fb6c16e4640 1 --2- 172.21.15.107:0/974783638 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6bc0d01c0 0x7fb6bc071ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:36.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.986+0000 7fb6c16e4640 1 -- 172.21.15.107:0/974783638 >> v2:172.21.15.195:3300/0 conn(0x7fb6bc072220 msgr2=0x7fb6bc0c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:36.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.986+0000 7fb6c16e4640 1 --2- 172.21.15.107:0/974783638 >> v2:172.21.15.195:3300/0 conn(0x7fb6bc072220 0x7fb6bc0c4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:36.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.986+0000 7fb6c16e4640 1 -- 172.21.15.107:0/974783638 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc15d710 con 0x7fb6bc154020 2023-12-04T05:47:36.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.986+0000 7fb6c16e4640 1 --2- 172.21.15.107:0/974783638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6bc154020 0x7fb6bc15a810 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0caa70 tx=0x7fb6b801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9830eb26067ad2a3 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:36.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.986+0000 7fb6b3fff640 1 -- 172.21.15.107:0/974783638 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b80a89b0 con 0x7fb6bc154020 2023-12-04T05:47:36.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.986+0000 7fb6b3fff640 1 -- 172.21.15.107:0/974783638 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b80a8b10 con 0x7fb6bc154020 2023-12-04T05:47:36.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.986+0000 7fb6b3fff640 1 -- 172.21.15.107:0/974783638 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b80a8ea0 con 0x7fb6bc154020 2023-12-04T05:47:36.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c3948640 1 -- 172.21.15.107:0/974783638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6bc154020 msgr2=0x7fb6bc15a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:36.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c3948640 1 --2- 172.21.15.107:0/974783638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6bc154020 0x7fb6bc15a810 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0caa70 tx=0x7fb6b801ecf0 comp rx=0 tx=0).stop 2023-12-04T05:47:36.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c3948640 1 -- 172.21.15.107:0/974783638 shutdown_connections 2023-12-04T05:47:36.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c3948640 1 --2- 172.21.15.107:0/974783638 >> v2:172.21.15.195:3300/0 conn(0x7fb6bc072220 0x7fb6bc0c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:36.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c3948640 1 --2- 172.21.15.107:0/974783638 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6bc0d01c0 0x7fb6bc071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:36.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c3948640 1 --2- 172.21.15.107:0/974783638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6bc154020 0x7fb6bc15a810 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:36.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c3948640 1 -- 172.21.15.107:0/974783638 >> 172.21.15.107:0/974783638 conn(0x7fb6bc06c6c0 msgr2=0x7fb6bc0cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:36.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c3948640 1 -- 172.21.15.107:0/974783638 shutdown_connections 2023-12-04T05:47:36.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c3948640 1 -- 172.21.15.107:0/974783638 wait complete. 2023-12-04T05:47:36.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c3948640 1 Processor -- start 2023-12-04T05:47:36.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c3948640 1 -- start start 2023-12-04T05:47:36.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c3948640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6bc072220 0x7fb6bc0c9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:36.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c3948640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6bc0d01c0 0x7fb6bc1505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:36.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c3948640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6bc154020 0x7fb6bc150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:36.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c3948640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc14fa30 con 0x7fb6bc0d01c0 2023-12-04T05:47:36.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c3948640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc0c99c0 con 0x7fb6bc072220 2023-12-04T05:47:36.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c3948640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc0c85b0 con 0x7fb6bc154020 2023-12-04T05:47:36.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c0ee3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6bc0d01c0 0x7fb6bc1505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:36.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c0ee3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6bc0d01c0 0x7fb6bc1505b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34550/0 (socket says 172.21.15.107:34550) 2023-12-04T05:47:36.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c0ee3640 1 -- 172.21.15.107:0/1276938166 learned_addr learned my addr 172.21.15.107:0/1276938166 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:36.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c1ee5640 1 --2- 172.21.15.107:0/1276938166 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6bc154020 0x7fb6bc150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:36.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c16e4640 1 --2- 172.21.15.107:0/1276938166 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6bc072220 0x7fb6bc0c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:36.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c0ee3640 1 -- 172.21.15.107:0/1276938166 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6bc154020 msgr2=0x7fb6bc150af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:36.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c0ee3640 1 --2- 172.21.15.107:0/1276938166 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6bc154020 0x7fb6bc150af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:36.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c0ee3640 1 -- 172.21.15.107:0/1276938166 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6bc072220 msgr2=0x7fb6bc0c9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:36.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c0ee3640 1 --2- 172.21.15.107:0/1276938166 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6bc072220 0x7fb6bc0c9100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:36.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c0ee3640 1 -- 172.21.15.107:0/1276938166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b80094d0 con 0x7fb6bc0d01c0 2023-12-04T05:47:36.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c0ee3640 1 --2- 172.21.15.107:0/1276938166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6bc0d01c0 0x7fb6bc1505b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac00d800 tx=0x7fb6ac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:36.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6b1ffb640 1 -- 172.21.15.107:0/1276938166 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac00b6c0 con 0x7fb6bc0d01c0 2023-12-04T05:47:36.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6b1ffb640 1 -- 172.21.15.107:0/1276938166 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6ac00b820 con 0x7fb6bc0d01c0 2023-12-04T05:47:36.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c3948640 1 -- 172.21.15.107:0/1276938166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6bc0c8850 con 0x7fb6bc0d01c0 2023-12-04T05:47:36.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6b1ffb640 1 -- 172.21.15.107:0/1276938166 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac00bb90 con 0x7fb6bc0d01c0 2023-12-04T05:47:36.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.990+0000 7fb6c3948640 1 -- 172.21.15.107:0/1276938166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6bc14fdb0 con 0x7fb6bc0d01c0 2023-12-04T05:47:36.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.994+0000 7fb6c3948640 1 -- 172.21.15.107:0/1276938166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6bc067870 con 0x7fb6bc0d01c0 2023-12-04T05:47:36.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.994+0000 7fb6b1ffb640 1 -- 172.21.15.107:0/1276938166 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fb6ac00bcf0 con 0x7fb6bc0d01c0 2023-12-04T05:47:36.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.994+0000 7fb6b1ffb640 1 --2- 172.21.15.107:0/1276938166 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb6a0066900 0x7fb6a0068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:36.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.994+0000 7fb6b1ffb640 1 -- 172.21.15.107:0/1276938166 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fb6ac084bc0 con 0x7fb6bc0d01c0 2023-12-04T05:47:36.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.994+0000 7fb6c16e4640 1 --2- 172.21.15.107:0/1276938166 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb6a0066900 0x7fb6a0068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:36.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.994+0000 7fb6c16e4640 1 --2- 172.21.15.107:0/1276938166 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb6a0066900 0x7fb6a0068dc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb6b80b2000 tx=0x7fb6b8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:36.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:36.998+0000 7fb6b1ffb640 1 -- 172.21.15.107:0/1276938166 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb6ac051ba0 con 0x7fb6bc0d01c0 2023-12-04T05:47:37.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:37.186+0000 7fb6c3948640 1 -- 172.21.15.107:0/1276938166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6bc0c9bc0 con 0x7fb6bc0d01c0 2023-12-04T05:47:37.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:37.186+0000 7fb6b1ffb640 1 -- 172.21.15.107:0/1276938166 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7fb6ac0551c0 con 0x7fb6bc0d01c0 2023-12-04T05:47:37.189 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:37.189 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:37.189 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:37.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:37.190+0000 7fb6c3948640 1 -- 172.21.15.107:0/1276938166 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb6a0066900 msgr2=0x7fb6a0068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:37.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:37.190+0000 7fb6c3948640 1 --2- 172.21.15.107:0/1276938166 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb6a0066900 0x7fb6a0068dc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb6b80b2000 tx=0x7fb6b8008040 comp rx=0 tx=0).stop 2023-12-04T05:47:37.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:37.190+0000 7fb6c3948640 1 -- 172.21.15.107:0/1276938166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6bc0d01c0 msgr2=0x7fb6bc1505b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:37.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:37.190+0000 7fb6c3948640 1 --2- 172.21.15.107:0/1276938166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6bc0d01c0 0x7fb6bc1505b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac00d800 tx=0x7fb6ac00dcc0 comp rx=0 tx=0).stop 2023-12-04T05:47:37.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:37.190+0000 7fb6c3948640 1 -- 172.21.15.107:0/1276938166 shutdown_connections 2023-12-04T05:47:37.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:37.190+0000 7fb6c3948640 1 --2- 172.21.15.107:0/1276938166 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb6a0066900 0x7fb6a0068dc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:37.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:37.190+0000 7fb6c3948640 1 --2- 172.21.15.107:0/1276938166 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb6bc154020 0x7fb6bc150af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:37.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:37.190+0000 7fb6c3948640 1 --2- 172.21.15.107:0/1276938166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6bc0d01c0 0x7fb6bc1505b0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:37.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:37.190+0000 7fb6c3948640 1 --2- 172.21.15.107:0/1276938166 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6bc072220 0x7fb6bc0c9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:37.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:37.190+0000 7fb6c3948640 1 -- 172.21.15.107:0/1276938166 >> 172.21.15.107:0/1276938166 conn(0x7fb6bc06c6c0 msgr2=0x7fb6bc151600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:37.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:37.190+0000 7fb6c3948640 1 -- 172.21.15.107:0/1276938166 shutdown_connections 2023-12-04T05:47:37.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:37.190+0000 7fb6c3948640 1 -- 172.21.15.107:0/1276938166 wait complete. 2023-12-04T05:47:37.209 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:47:39.210 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:39.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.338+0000 7f7803699640 1 Processor -- start 2023-12-04T05:47:39.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.338+0000 7f7803699640 1 -- start start 2023-12-04T05:47:39.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.338+0000 7f7803699640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f77fc0724f0 0x7f77fc158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:39.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.338+0000 7f7803699640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77fc158da0 con 0x7f77fc0c56f0 2023-12-04T05:47:39.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.338+0000 7f7803699640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77fc1583d0 con 0x7f77fc072110 2023-12-04T05:47:39.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.338+0000 7f7803699640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77fc0c6170 con 0x7f77fc0724f0 2023-12-04T05:47:39.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.338+0000 7f7800c34640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f77fc0724f0 0x7f77fc158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:39.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.338+0000 7f7801c36640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f77fc0c56f0 0x7f77fc140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41922/0 (socket says 172.21.15.107:41922) 2023-12-04T05:47:39.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.338+0000 7f7800c34640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f77fc0724f0 0x7f77fc158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38176/0 (socket says 172.21.15.107:38176) 2023-12-04T05:47:39.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.338+0000 7f7801c36640 1 -- 172.21.15.107:0/84233858 learned_addr learned my addr 172.21.15.107:0/84233858 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:47:39.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.338+0000 7f7800c34640 1 -- 172.21.15.107:0/84233858 >> v1:172.21.15.195:6789/0 conn(0x7f77fc072110 legacy=0x7f77fc0c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:39.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.338+0000 7f77f3fff640 1 -- 172.21.15.107:0/84233858 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027174336 0 0) 0x7f77f40014a0 con 0x7f77fc0c56f0 2023-12-04T05:47:39.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.338+0000 7f7800c34640 1 -- 172.21.15.107:0/84233858 >> v1:172.21.15.107:6789/0 conn(0x7f77fc0c56f0 legacy=0x7f77fc140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:39.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.338+0000 7f7800c34640 1 -- 172.21.15.107:0/84233858 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77fc157da0 con 0x7f77fc0724f0 2023-12-04T05:47:39.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7800c34640 1 --2- 172.21.15.107:0/84233858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77fc0724f0 0x7f77fc158860 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f77ec0097c0 tx=0x7f77ec01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b78710f5aabb928e server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:39.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f77f3fff640 1 -- 172.21.15.107:0/84233858 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ec0a8970 con 0x7f77fc0724f0 2023-12-04T05:47:39.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f77f3fff640 1 -- 172.21.15.107:0/84233858 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77ec0a8ad0 con 0x7f77fc0724f0 2023-12-04T05:47:39.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f77f3fff640 1 -- 172.21.15.107:0/84233858 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ec0a8e60 con 0x7f77fc0724f0 2023-12-04T05:47:39.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7803699640 1 -- 172.21.15.107:0/84233858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77fc0724f0 msgr2=0x7f77fc158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:39.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7803699640 1 --2- 172.21.15.107:0/84233858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77fc0724f0 0x7f77fc158860 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f77ec0097c0 tx=0x7f77ec01ec70 comp rx=0 tx=0).stop 2023-12-04T05:47:39.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7803699640 1 -- 172.21.15.107:0/84233858 shutdown_connections 2023-12-04T05:47:39.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7803699640 1 --2- 172.21.15.107:0/84233858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77fc0724f0 0x7f77fc158860 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:39.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7803699640 1 -- 172.21.15.107:0/84233858 >> 172.21.15.107:0/84233858 conn(0x7f77fc0c7ff0 msgr2=0x7f77fc0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:39.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7803699640 1 -- 172.21.15.107:0/84233858 shutdown_connections 2023-12-04T05:47:39.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7803699640 1 -- 172.21.15.107:0/84233858 wait complete. 2023-12-04T05:47:39.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7803699640 1 Processor -- start 2023-12-04T05:47:39.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7803699640 1 -- start start 2023-12-04T05:47:39.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7803699640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f77fc072110 0x7f77fc0caa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:39.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7803699640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77fc0724f0 0x7f77fc158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:39.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7803699640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f77fc0c56f0 0x7f77fc1594a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:39.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7803699640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc1ee1c0 con 0x7f77fc0724f0 2023-12-04T05:47:39.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7803699640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc1ed580 con 0x7f77fc0c56f0 2023-12-04T05:47:39.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7803699640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc1ed820 con 0x7f77fc072110 2023-12-04T05:47:39.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7800c34640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77fc0724f0 0x7f77fc158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:39.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7800c34640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77fc0724f0 0x7f77fc158f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38178/0 (socket says 172.21.15.107:38178) 2023-12-04T05:47:39.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7800c34640 1 -- 172.21.15.107:0/1365357366 learned_addr learned my addr 172.21.15.107:0/1365357366 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:39.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7800c34640 1 -- 172.21.15.107:0/1365357366 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f77fc072110 msgr2=0x7f77fc0caa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:39.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7801435640 1 --2- 172.21.15.107:0/1365357366 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f77fc072110 0x7f77fc0caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:39.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7801c36640 1 --2- 172.21.15.107:0/1365357366 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f77fc0c56f0 0x7f77fc1594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:39.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7800c34640 1 --2- 172.21.15.107:0/1365357366 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f77fc072110 0x7f77fc0caa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:39.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7800c34640 1 -- 172.21.15.107:0/1365357366 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f77fc0c56f0 msgr2=0x7f77fc1594a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:39.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7800c34640 1 --2- 172.21.15.107:0/1365357366 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f77fc0c56f0 0x7f77fc1594a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:39.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7800c34640 1 -- 172.21.15.107:0/1365357366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ec009420 con 0x7f77fc0724f0 2023-12-04T05:47:39.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.342+0000 7f7800c34640 1 --2- 172.21.15.107:0/1365357366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77fc0724f0 0x7f77fc158f60 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f77ec01b880 tx=0x7f77ec01ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:39.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.346+0000 7f77f1ffb640 1 -- 172.21.15.107:0/1365357366 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ec0a8970 con 0x7f77fc0724f0 2023-12-04T05:47:39.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.346+0000 7f77f1ffb640 1 -- 172.21.15.107:0/1365357366 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77ec0b9410 con 0x7f77fc0724f0 2023-12-04T05:47:39.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.346+0000 7f7803699640 1 -- 172.21.15.107:0/1365357366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77fc1edb00 con 0x7f77fc0724f0 2023-12-04T05:47:39.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.346+0000 7f77f1ffb640 1 -- 172.21.15.107:0/1365357366 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ec0b97a0 con 0x7f77fc0724f0 2023-12-04T05:47:39.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.346+0000 7f7803699640 1 -- 172.21.15.107:0/1365357366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77fc0caf50 con 0x7f77fc0724f0 2023-12-04T05:47:39.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.346+0000 7f7803699640 1 -- 172.21.15.107:0/1365357366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77fc067870 con 0x7f77fc0724f0 2023-12-04T05:47:39.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.346+0000 7f77f1ffb640 1 -- 172.21.15.107:0/1365357366 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f77ec0b7040 con 0x7f77fc0724f0 2023-12-04T05:47:39.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.346+0000 7f77f1ffb640 1 --2- 172.21.15.107:0/1365357366 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f77e00668d0 0x7f77e0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:39.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.346+0000 7f7801435640 1 --2- 172.21.15.107:0/1365357366 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f77e00668d0 0x7f77e0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:39.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.346+0000 7f77f1ffb640 1 -- 172.21.15.107:0/1365357366 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f77ec122300 con 0x7f77fc0724f0 2023-12-04T05:47:39.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.346+0000 7f7801435640 1 --2- 172.21.15.107:0/1365357366 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f77e00668d0 0x7f77e0068d90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f77f8000f70 tx=0x7f77f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:39.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.350+0000 7f77f1ffb640 1 -- 172.21.15.107:0/1365357366 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f77ec0ef370 con 0x7f77fc0724f0 2023-12-04T05:47:39.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.534+0000 7f7803699640 1 -- 172.21.15.107:0/1365357366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77fc150e80 con 0x7f77fc0724f0 2023-12-04T05:47:39.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.534+0000 7f77f1ffb640 1 -- 172.21.15.107:0/1365357366 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f77ec0f2990 con 0x7f77fc0724f0 2023-12-04T05:47:39.537 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:39.537 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:39.537 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:39.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.538+0000 7f7803699640 1 -- 172.21.15.107:0/1365357366 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f77e00668d0 msgr2=0x7f77e0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:39.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.538+0000 7f7803699640 1 --2- 172.21.15.107:0/1365357366 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f77e00668d0 0x7f77e0068d90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f77f8000f70 tx=0x7f77f8009040 comp rx=0 tx=0).stop 2023-12-04T05:47:39.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.538+0000 7f7803699640 1 -- 172.21.15.107:0/1365357366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77fc0724f0 msgr2=0x7f77fc158f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:39.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.538+0000 7f7803699640 1 --2- 172.21.15.107:0/1365357366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77fc0724f0 0x7f77fc158f60 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f77ec01b880 tx=0x7f77ec01ba10 comp rx=0 tx=0).stop 2023-12-04T05:47:39.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.538+0000 7f7803699640 1 -- 172.21.15.107:0/1365357366 shutdown_connections 2023-12-04T05:47:39.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.538+0000 7f7803699640 1 --2- 172.21.15.107:0/1365357366 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f77e00668d0 0x7f77e0068d90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:39.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.538+0000 7f7803699640 1 --2- 172.21.15.107:0/1365357366 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f77fc0c56f0 0x7f77fc1594a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:39.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.538+0000 7f7803699640 1 --2- 172.21.15.107:0/1365357366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77fc0724f0 0x7f77fc158f60 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:39.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.538+0000 7f7803699640 1 --2- 172.21.15.107:0/1365357366 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f77fc072110 0x7f77fc0caa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:39.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.538+0000 7f7803699640 1 -- 172.21.15.107:0/1365357366 >> 172.21.15.107:0/1365357366 conn(0x7f77fc0c7ff0 msgr2=0x7f77fc156e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:39.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.538+0000 7f7803699640 1 -- 172.21.15.107:0/1365357366 shutdown_connections 2023-12-04T05:47:39.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:39.538+0000 7f7803699640 1 -- 172.21.15.107:0/1365357366 wait complete. 2023-12-04T05:47:39.555 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 50, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-12-04T05:39:59.032019+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_4288": { "addr": "172.21.15.195:6837/3412865477", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6835", "nonce": 3412865477, "type": "v2" }, { "addr": "172.21.15.195:6837", "nonce": 3412865477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_5190": { "addr": "172.21.15.107:6836/2928623624", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 2928623624, "type": "v2" }, { "addr": "172.21.15.107:6836", "nonce": 2928623624, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5190, "incarnation": 45, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:rejoin", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-12-04T05:46:11.040247+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2, 3, 4 ], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 5190 } } } ], "standbys": [ { "addr": "172.21.15.107:6837/2752497349", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6835", "nonce": 2752497349, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 2752497349, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 49, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5292, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-12-04T05:47:40.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~119s 2023-12-04T05:47:40.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~233s 2023-12-04T05:47:40.788 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~274s 2023-12-04T05:47:41.556 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:41.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.682+0000 7f179dea2640 1 Processor -- start 2023-12-04T05:47:41.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f179dea2640 1 -- start start 2023-12-04T05:47:41.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f179dea2640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1798151180 0x7f17980c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:41.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f179dea2640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f17980714f0 0x7f17980718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:41.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f179dea2640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17980700c0 con 0x7f1798070380 2023-12-04T05:47:41.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f179dea2640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1798156df0 con 0x7f17980714f0 2023-12-04T05:47:41.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f179dea2640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17981570d0 con 0x7f1798151180 2023-12-04T05:47:41.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f1797fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1798070380 0x7f1798070790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41926/0 (socket says 172.21.15.107:41926) 2023-12-04T05:47:41.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f1797fff640 1 -- 172.21.15.107:0/2189885241 learned_addr learned my addr 172.21.15.107:0/2189885241 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:47:41.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f1796ffd640 1 --2- 172.21.15.107:0/2189885241 >> v2:172.21.15.195:3300/0 conn(0x7f17980714f0 0x7f17980718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17977fe640 1 --2- 172.21.15.107:0/2189885241 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1798151180 0x7f17980c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17967fc640 1 -- 172.21.15.107:0/2189885241 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650883555 0 0) 0x7f17980700c0 con 0x7f1798070380 2023-12-04T05:47:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17967fc640 1 -- 172.21.15.107:0/2189885241 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1780003530 con 0x7f1798070380 2023-12-04T05:47:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17967fc640 1 -- 172.21.15.107:0/2189885241 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 559847460 0 0) 0x7f1780003530 con 0x7f1798070380 2023-12-04T05:47:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17967fc640 1 -- 172.21.15.107:0/2189885241 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1798151180 msgr2=0x7f17980c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17967fc640 1 --2- 172.21.15.107:0/2189885241 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1798151180 0x7f17980c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17967fc640 1 -- 172.21.15.107:0/2189885241 >> v2:172.21.15.195:3300/0 conn(0x7f17980714f0 msgr2=0x7f17980718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17967fc640 1 --2- 172.21.15.107:0/2189885241 >> v2:172.21.15.195:3300/0 conn(0x7f17980714f0 0x7f17980718f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17967fc640 1 -- 172.21.15.107:0/2189885241 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17981573b0 con 0x7f1798070380 2023-12-04T05:47:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17967fc640 1 -- 172.21.15.107:0/2189885241 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f1788001a00 con 0x7f1798070380 2023-12-04T05:47:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17967fc640 1 -- 172.21.15.107:0/2189885241 >> v1:172.21.15.107:6789/0 conn(0x7f1798070380 legacy=0x7f1798070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17967fc640 1 --2- 172.21.15.107:0/2189885241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1780003980 0x7f1780003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17967fc640 1 --2- 172.21.15.107:0/2189885241 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1798151180 0x7f17800047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17967fc640 1 --2- 172.21.15.107:0/2189885241 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1780004d10 0x7f1780005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17967fc640 1 -- 172.21.15.107:0/2189885241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1798156df0 con 0x7f1780003980 2023-12-04T05:47:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17967fc640 1 -- 172.21.15.107:0/2189885241 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17981570d0 con 0x7f1780004d10 2023-12-04T05:47:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17967fc640 1 -- 172.21.15.107:0/2189885241 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178000b990 con 0x7f1798151180 2023-12-04T05:47:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17977fe640 1 --2- 172.21.15.107:0/2189885241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1780003980 0x7f1780003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f1796ffd640 1 --2- 172.21.15.107:0/2189885241 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1798151180 0x7f17800047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f1797fff640 1 --2- 172.21.15.107:0/2189885241 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1780004d10 0x7f1780005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17977fe640 1 -- 172.21.15.107:0/2189885241 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1798151180 msgr2=0x7f17800047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17977fe640 1 --2- 172.21.15.107:0/2189885241 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1798151180 0x7f17800047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17977fe640 1 -- 172.21.15.107:0/2189885241 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1780004d10 msgr2=0x7f1780005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17977fe640 1 --2- 172.21.15.107:0/2189885241 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1780004d10 0x7f1780005190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f17977fe640 1 -- 172.21.15.107:0/2189885241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1780003530 con 0x7f1780003980 2023-12-04T05:47:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.686+0000 7f1796ffd640 1 --2- 172.21.15.107:0/2189885241 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1798151180 0x7f17800047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:47:41.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.690+0000 7f17977fe640 1 --2- 172.21.15.107:0/2189885241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1780003980 0x7f1780003da0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f1790009ae0 tx=0x7f179001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=272138afcf9cdcaa server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:41.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.690+0000 7f17967fc640 1 -- 172.21.15.107:0/2189885241 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17900a5640 con 0x7f1780003980 2023-12-04T05:47:41.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.690+0000 7f17967fc640 1 -- 172.21.15.107:0/2189885241 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17900a57a0 con 0x7f1780003980 2023-12-04T05:47:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.690+0000 7f17967fc640 1 -- 172.21.15.107:0/2189885241 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17900a5b50 con 0x7f1780003980 2023-12-04T05:47:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.690+0000 7f179dea2640 1 -- 172.21.15.107:0/2189885241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1780003980 msgr2=0x7f1780003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.690+0000 7f179dea2640 1 --2- 172.21.15.107:0/2189885241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1780003980 0x7f1780003da0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f1790009ae0 tx=0x7f179001ca50 comp rx=0 tx=0).stop 2023-12-04T05:47:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.690+0000 7f1797fff640 1 -- 172.21.15.107:0/2189885241 reap_dead start 2023-12-04T05:47:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.690+0000 7f179dea2640 1 -- 172.21.15.107:0/2189885241 shutdown_connections 2023-12-04T05:47:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.690+0000 7f179dea2640 1 -- 172.21.15.107:0/2189885241 >> 172.21.15.107:0/2189885241 conn(0x7f17980c7ff0 msgr2=0x7f17980c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.690+0000 7f179dea2640 1 -- 172.21.15.107:0/2189885241 shutdown_connections 2023-12-04T05:47:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.690+0000 7f179dea2640 1 -- 172.21.15.107:0/2189885241 wait complete. 2023-12-04T05:47:41.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.690+0000 7f179dea2640 1 Processor -- start 2023-12-04T05:47:41.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.690+0000 7f179dea2640 1 -- start start 2023-12-04T05:47:41.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.690+0000 7f179dea2640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1780003980 0x7f17981e46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:41.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.690+0000 7f179dea2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17980c5370 0x7f17981f0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:41.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.690+0000 7f179dea2640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1798156810 0x7f17981f07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:41.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.690+0000 7f179dea2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17980c3610 con 0x7f17980c5370 2023-12-04T05:47:41.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.690+0000 7f179dea2640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179814e760 con 0x7f1798156810 2023-12-04T05:47:41.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f179dea2640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f179814c0d0 con 0x7f1780003980 2023-12-04T05:47:41.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f1796ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17980c5370 0x7f17981f0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:41.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f1796ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17980c5370 0x7f17981f0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38194/0 (socket says 172.21.15.107:38194) 2023-12-04T05:47:41.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f1796ffd640 1 -- 172.21.15.107:0/1684259495 learned_addr learned my addr 172.21.15.107:0/1684259495 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:41.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f17977fe640 1 --2- 172.21.15.107:0/1684259495 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1780003980 0x7f17981e46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:41.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f1797fff640 1 --2- 172.21.15.107:0/1684259495 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1798156810 0x7f17981f07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:41.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f1796ffd640 1 -- 172.21.15.107:0/1684259495 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1780003980 msgr2=0x7f17981e46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:41.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f1796ffd640 1 --2- 172.21.15.107:0/1684259495 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1780003980 0x7f17981e46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:41.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f1796ffd640 1 -- 172.21.15.107:0/1684259495 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1798156810 msgr2=0x7f17981f07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:41.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f1796ffd640 1 --2- 172.21.15.107:0/1684259495 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1798156810 0x7f17981f07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:41.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f1796ffd640 1 -- 172.21.15.107:0/1684259495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f179814b330 con 0x7f17980c5370 2023-12-04T05:47:41.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f1797fff640 1 --2- 172.21.15.107:0/1684259495 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1798156810 0x7f17981f07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:47:41.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f1796ffd640 1 --2- 172.21.15.107:0/1684259495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17980c5370 0x7f17981f0280 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f178c010950 tx=0x7f178c010e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:41.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f177ffff640 1 -- 172.21.15.107:0/1684259495 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178c00ec20 con 0x7f17980c5370 2023-12-04T05:47:41.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f177ffff640 1 -- 172.21.15.107:0/1684259495 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f178c00ed80 con 0x7f17980c5370 2023-12-04T05:47:41.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f179dea2640 1 -- 172.21.15.107:0/1684259495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f179814b5d0 con 0x7f17980c5370 2023-12-04T05:47:41.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f177ffff640 1 -- 172.21.15.107:0/1684259495 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178c004220 con 0x7f17980c5370 2023-12-04T05:47:41.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f179dea2640 1 -- 172.21.15.107:0/1684259495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f179814bb10 con 0x7f17980c5370 2023-12-04T05:47:41.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f179dea2640 1 -- 172.21.15.107:0/1684259495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1798067870 con 0x7f17980c5370 2023-12-04T05:47:41.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f177ffff640 1 -- 172.21.15.107:0/1684259495 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f178c004380 con 0x7f17980c5370 2023-12-04T05:47:41.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f177ffff640 1 --2- 172.21.15.107:0/1684259495 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1784066900 0x7f1784068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:41.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f177ffff640 1 -- 172.21.15.107:0/1684259495 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f178c084460 con 0x7f17980c5370 2023-12-04T05:47:41.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f17977fe640 1 --2- 172.21.15.107:0/1684259495 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1784066900 0x7f1784068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:41.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.694+0000 7f17977fe640 1 --2- 172.21.15.107:0/1684259495 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1784066900 0x7f1784068dc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f17900af000 tx=0x7f17900046d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:41.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.698+0000 7f177ffff640 1 -- 172.21.15.107:0/1684259495 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f178c051560 con 0x7f17980c5370 2023-12-04T05:47:41.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.886+0000 7f179dea2640 1 -- 172.21.15.107:0/1684259495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1798071e00 con 0x7f17980c5370 2023-12-04T05:47:41.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.886+0000 7f177ffff640 1 -- 172.21.15.107:0/1684259495 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f178c054b80 con 0x7f17980c5370 2023-12-04T05:47:41.889 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:41.889 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:41.889 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:41.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.890+0000 7f179dea2640 1 -- 172.21.15.107:0/1684259495 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1784066900 msgr2=0x7f1784068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:41.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.890+0000 7f179dea2640 1 --2- 172.21.15.107:0/1684259495 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1784066900 0x7f1784068dc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f17900af000 tx=0x7f17900046d0 comp rx=0 tx=0).stop 2023-12-04T05:47:41.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.890+0000 7f179dea2640 1 -- 172.21.15.107:0/1684259495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17980c5370 msgr2=0x7f17981f0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:41.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.890+0000 7f179dea2640 1 --2- 172.21.15.107:0/1684259495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17980c5370 0x7f17981f0280 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f178c010950 tx=0x7f178c010e10 comp rx=0 tx=0).stop 2023-12-04T05:47:41.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.890+0000 7f179dea2640 1 -- 172.21.15.107:0/1684259495 shutdown_connections 2023-12-04T05:47:41.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.890+0000 7f179dea2640 1 --2- 172.21.15.107:0/1684259495 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1784066900 0x7f1784068dc0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:41.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.890+0000 7f179dea2640 1 --2- 172.21.15.107:0/1684259495 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1798156810 0x7f17981f07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:41.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.890+0000 7f179dea2640 1 --2- 172.21.15.107:0/1684259495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17980c5370 0x7f17981f0280 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:41.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.890+0000 7f179dea2640 1 --2- 172.21.15.107:0/1684259495 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1780003980 0x7f17981e46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:41.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.890+0000 7f179dea2640 1 -- 172.21.15.107:0/1684259495 >> 172.21.15.107:0/1684259495 conn(0x7f17980c7ff0 msgr2=0x7f1798070760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:41.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.890+0000 7f179dea2640 1 -- 172.21.15.107:0/1684259495 shutdown_connections 2023-12-04T05:47:41.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:41.890+0000 7f179dea2640 1 -- 172.21.15.107:0/1684259495 wait complete. 2023-12-04T05:47:41.908 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:47:43.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:44.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.038+0000 7f6e89c27640 1 Processor -- start 2023-12-04T05:47:44.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.038+0000 7f6e89c27640 1 -- start start 2023-12-04T05:47:44.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e89c27640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6e840724f0 0x7f6e84158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:44.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e89c27640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e84158da0 con 0x7f6e84072110 2023-12-04T05:47:44.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e89c27640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e841583d0 con 0x7f6e840c56f0 2023-12-04T05:47:44.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e89c27640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e840c6170 con 0x7f6e840724f0 2023-12-04T05:47:44.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e837fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6e84072110 0x7f6e840c7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41928/0 (socket says 172.21.15.107:41928) 2023-12-04T05:47:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e837fe640 1 -- 172.21.15.107:0/568959974 learned_addr learned my addr 172.21.15.107:0/568959974 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:47:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e827fc640 1 -- 172.21.15.107:0/568959974 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639355309 0 0) 0x7f6e74001120 con 0x7f6e84072110 2023-12-04T05:47:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e82ffd640 1 --2- 172.21.15.107:0/568959974 >> v2:172.21.15.195:3300/0 conn(0x7f6e840724f0 0x7f6e84158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e827fc640 1 -- 172.21.15.107:0/568959974 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e6c003530 con 0x7f6e84072110 2023-12-04T05:47:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e827fc640 1 -- 172.21.15.107:0/568959974 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 309577115 0 0) 0x7f6e6c003530 con 0x7f6e84072110 2023-12-04T05:47:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e827fc640 1 -- 172.21.15.107:0/568959974 >> v2:172.21.15.195:3300/0 conn(0x7f6e840724f0 msgr2=0x7f6e84158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e827fc640 1 --2- 172.21.15.107:0/568959974 >> v2:172.21.15.195:3300/0 conn(0x7f6e840724f0 0x7f6e84158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e827fc640 1 -- 172.21.15.107:0/568959974 >> v1:172.21.15.195:6789/0 conn(0x7f6e840c56f0 legacy=0x7f6e84140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e827fc640 1 -- 172.21.15.107:0/568959974 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e84157da0 con 0x7f6e84072110 2023-12-04T05:47:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e827fc640 1 -- 172.21.15.107:0/568959974 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f6e74001810 con 0x7f6e84072110 2023-12-04T05:47:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e82ffd640 1 --2- 172.21.15.107:0/568959974 >> v2:172.21.15.195:3300/0 conn(0x7f6e840724f0 0x7f6e84158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:47:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e827fc640 1 -- 172.21.15.107:0/568959974 >> v1:172.21.15.107:6789/0 conn(0x7f6e84072110 legacy=0x7f6e840c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e827fc640 1 --2- 172.21.15.107:0/568959974 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e6c003980 0x7f6e6c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e827fc640 1 --2- 172.21.15.107:0/568959974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e6c0047d0 0x7f6e6c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e827fc640 1 --2- 172.21.15.107:0/568959974 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6e6c005190 0x7f6e6c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:44.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e827fc640 1 -- 172.21.15.107:0/568959974 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840c6170 con 0x7f6e6c0047d0 2023-12-04T05:47:44.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e827fc640 1 -- 172.21.15.107:0/568959974 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c00c0c0 con 0x7f6e6c003980 2023-12-04T05:47:44.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e827fc640 1 -- 172.21.15.107:0/568959974 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c00c280 con 0x7f6e6c005190 2023-12-04T05:47:44.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e82ffd640 1 --2- 172.21.15.107:0/568959974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e6c0047d0 0x7f6e6c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:44.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e837fe640 1 --2- 172.21.15.107:0/568959974 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e6c003980 0x7f6e6c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:44.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e83fff640 1 --2- 172.21.15.107:0/568959974 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6e6c005190 0x7f6e6c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:44.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e82ffd640 1 -- 172.21.15.107:0/568959974 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6e6c005190 msgr2=0x7f6e6c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:44.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e82ffd640 1 --2- 172.21.15.107:0/568959974 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6e6c005190 0x7f6e6c00b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:44.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e82ffd640 1 -- 172.21.15.107:0/568959974 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e6c003980 msgr2=0x7f6e6c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:44.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e82ffd640 1 --2- 172.21.15.107:0/568959974 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e6c003980 0x7f6e6c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:44.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e82ffd640 1 -- 172.21.15.107:0/568959974 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c003530 con 0x7f6e6c0047d0 2023-12-04T05:47:44.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.042+0000 7f6e83fff640 1 -- 172.21.15.107:0/568959974 reap_dead start 2023-12-04T05:47:44.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e82ffd640 1 --2- 172.21.15.107:0/568959974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e6c0047d0 0x7f6e6c004c50 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f6e780098a0 tx=0x7f6e7801c570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=958257fc6ab60964 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:44.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e827fc640 1 -- 172.21.15.107:0/568959974 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e780a9680 con 0x7f6e6c0047d0 2023-12-04T05:47:44.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e827fc640 1 -- 172.21.15.107:0/568959974 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e780a97e0 con 0x7f6e6c0047d0 2023-12-04T05:47:44.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e827fc640 1 -- 172.21.15.107:0/568959974 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e780a9b70 con 0x7f6e6c0047d0 2023-12-04T05:47:44.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e89c27640 1 -- 172.21.15.107:0/568959974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e6c0047d0 msgr2=0x7f6e6c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:44.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e89c27640 1 --2- 172.21.15.107:0/568959974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e6c0047d0 0x7f6e6c004c50 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f6e780098a0 tx=0x7f6e7801c570 comp rx=0 tx=0).stop 2023-12-04T05:47:44.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e89c27640 1 -- 172.21.15.107:0/568959974 shutdown_connections 2023-12-04T05:47:44.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e89c27640 1 --2- 172.21.15.107:0/568959974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e6c0047d0 0x7f6e6c004c50 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:44.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e89c27640 1 -- 172.21.15.107:0/568959974 >> 172.21.15.107:0/568959974 conn(0x7f6e840c7ff0 msgr2=0x7f6e840c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:44.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e89c27640 1 -- 172.21.15.107:0/568959974 shutdown_connections 2023-12-04T05:47:44.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e89c27640 1 -- 172.21.15.107:0/568959974 wait complete. 2023-12-04T05:47:44.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e89c27640 1 Processor -- start 2023-12-04T05:47:44.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e89c27640 1 -- start start 2023-12-04T05:47:44.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e89c27640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6e6c0047d0 0x7f6e841528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:44.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e89c27640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e841578c0 0x7f6e841549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:44.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e89c27640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e84157270 0x7f6e84154ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:44.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e89c27640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840716a0 con 0x7f6e841578c0 2023-12-04T05:47:44.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e89c27640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8406e060 con 0x7f6e84157270 2023-12-04T05:47:44.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e89c27640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840718b0 con 0x7f6e6c0047d0 2023-12-04T05:47:44.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e82ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e841578c0 0x7f6e841549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:44.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e82ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e841578c0 0x7f6e841549b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38208/0 (socket says 172.21.15.107:38208) 2023-12-04T05:47:44.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e82ffd640 1 -- 172.21.15.107:0/4159065513 learned_addr learned my addr 172.21.15.107:0/4159065513 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:44.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e837fe640 1 --2- 172.21.15.107:0/4159065513 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6e6c0047d0 0x7f6e841528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:44.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e83fff640 1 --2- 172.21.15.107:0/4159065513 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e84157270 0x7f6e84154ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:44.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e82ffd640 1 -- 172.21.15.107:0/4159065513 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6e6c0047d0 msgr2=0x7f6e841528f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:44.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e82ffd640 1 --2- 172.21.15.107:0/4159065513 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6e6c0047d0 0x7f6e841528f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:44.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e82ffd640 1 -- 172.21.15.107:0/4159065513 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e84157270 msgr2=0x7f6e84154ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:44.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e82ffd640 1 --2- 172.21.15.107:0/4159065513 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e84157270 0x7f6e84154ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:44.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e82ffd640 1 -- 172.21.15.107:0/4159065513 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e84071b90 con 0x7f6e841578c0 2023-12-04T05:47:44.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e82ffd640 1 --2- 172.21.15.107:0/4159065513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e841578c0 0x7f6e841549b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6e7801c3a0 tx=0x7f6e6c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:44.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e6bfff640 1 -- 172.21.15.107:0/4159065513 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e780b7070 con 0x7f6e841578c0 2023-12-04T05:47:44.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e6bfff640 1 -- 172.21.15.107:0/4159065513 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e780bc420 con 0x7f6e841578c0 2023-12-04T05:47:44.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e89c27640 1 -- 172.21.15.107:0/4159065513 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e8406e300 con 0x7f6e841578c0 2023-12-04T05:47:44.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.046+0000 7f6e6bfff640 1 -- 172.21.15.107:0/4159065513 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e780bc7d0 con 0x7f6e841578c0 2023-12-04T05:47:44.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.050+0000 7f6e89c27640 1 -- 172.21.15.107:0/4159065513 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e840c3fe0 con 0x7f6e841578c0 2023-12-04T05:47:44.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.050+0000 7f6e89c27640 1 -- 172.21.15.107:0/4159065513 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e84067870 con 0x7f6e841578c0 2023-12-04T05:47:44.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.050+0000 7f6e6bfff640 1 -- 172.21.15.107:0/4159065513 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f6e780b13f0 con 0x7f6e841578c0 2023-12-04T05:47:44.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.050+0000 7f6e6bfff640 1 --2- 172.21.15.107:0/4159065513 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6e70066900 0x7f6e70068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:44.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.050+0000 7f6e6bfff640 1 -- 172.21.15.107:0/4159065513 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f6e7811ee10 con 0x7f6e841578c0 2023-12-04T05:47:44.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.050+0000 7f6e837fe640 1 --2- 172.21.15.107:0/4159065513 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6e70066900 0x7f6e70068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:44.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.050+0000 7f6e837fe640 1 --2- 172.21.15.107:0/4159065513 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6e70066900 0x7f6e70068dc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6e74001060 tx=0x7f6e74009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:44.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.054+0000 7f6e6bfff640 1 -- 172.21.15.107:0/4159065513 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6e780ebdf0 con 0x7f6e841578c0 2023-12-04T05:47:44.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.242+0000 7f6e89c27640 1 -- 172.21.15.107:0/4159065513 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e840c67e0 con 0x7f6e841578c0 2023-12-04T05:47:44.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.242+0000 7f6e6bfff640 1 -- 172.21.15.107:0/4159065513 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f6e7800f040 con 0x7f6e841578c0 2023-12-04T05:47:44.244 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:44.244 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:44.245 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:44.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.242+0000 7f6e89c27640 1 -- 172.21.15.107:0/4159065513 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6e70066900 msgr2=0x7f6e70068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:44.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.242+0000 7f6e89c27640 1 --2- 172.21.15.107:0/4159065513 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6e70066900 0x7f6e70068dc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6e74001060 tx=0x7f6e74009040 comp rx=0 tx=0).stop 2023-12-04T05:47:44.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.246+0000 7f6e89c27640 1 -- 172.21.15.107:0/4159065513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e841578c0 msgr2=0x7f6e841549b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:44.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.246+0000 7f6e89c27640 1 --2- 172.21.15.107:0/4159065513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e841578c0 0x7f6e841549b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6e7801c3a0 tx=0x7f6e6c005800 comp rx=0 tx=0).stop 2023-12-04T05:47:44.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.246+0000 7f6e89c27640 1 -- 172.21.15.107:0/4159065513 shutdown_connections 2023-12-04T05:47:44.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.246+0000 7f6e89c27640 1 --2- 172.21.15.107:0/4159065513 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6e70066900 0x7f6e70068dc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:44.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.246+0000 7f6e89c27640 1 --2- 172.21.15.107:0/4159065513 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e84157270 0x7f6e84154ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:44.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.246+0000 7f6e89c27640 1 --2- 172.21.15.107:0/4159065513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e841578c0 0x7f6e841549b0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:44.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.246+0000 7f6e89c27640 1 --2- 172.21.15.107:0/4159065513 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6e6c0047d0 0x7f6e841528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:44.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.246+0000 7f6e89c27640 1 -- 172.21.15.107:0/4159065513 >> 172.21.15.107:0/4159065513 conn(0x7f6e840c7ff0 msgr2=0x7f6e84155c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:44.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.246+0000 7f6e89c27640 1 -- 172.21.15.107:0/4159065513 shutdown_connections 2023-12-04T05:47:44.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:44.246+0000 7f6e89c27640 1 -- 172.21.15.107:0/4159065513 wait complete. 2023-12-04T05:47:44.263 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:47:46.264 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:46.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f042bea3640 1 Processor -- start 2023-12-04T05:47:46.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f042bea3640 1 -- start start 2023-12-04T05:47:46.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f042bea3640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f042415e6b0 0x7f042415ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:46.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f042bea3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04240c5ef0 con 0x7f04240c8c10 2023-12-04T05:47:46.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f042bea3640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04240c25e0 con 0x7f04241588e0 2023-12-04T05:47:46.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f042bea3640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04240c6290 con 0x7f042415e6b0 2023-12-04T05:47:46.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f042943e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f04240c8c10 0x7f04241581d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41936/0 (socket says 172.21.15.107:41936) 2023-12-04T05:47:46.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f042943e640 1 -- 172.21.15.107:0/2714767467 learned_addr learned my addr 172.21.15.107:0/2714767467 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:47:46.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f0429c3f640 1 --2- 172.21.15.107:0/2714767467 >> v2:172.21.15.195:3300/0 conn(0x7f042415e6b0 0x7f042415ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f0428c3d640 1 -- 172.21.15.107:0/2714767467 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653909728 0 0) 0x7f04240c5ef0 con 0x7f04240c8c10 2023-12-04T05:47:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f0428c3d640 1 -- 172.21.15.107:0/2714767467 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0410003530 con 0x7f04240c8c10 2023-12-04T05:47:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f0428c3d640 1 -- 172.21.15.107:0/2714767467 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 130811586 0 0) 0x7f0410003530 con 0x7f04240c8c10 2023-12-04T05:47:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f0428c3d640 1 -- 172.21.15.107:0/2714767467 >> v2:172.21.15.195:3300/0 conn(0x7f042415e6b0 msgr2=0x7f042415ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f0428c3d640 1 --2- 172.21.15.107:0/2714767467 >> v2:172.21.15.195:3300/0 conn(0x7f042415e6b0 0x7f042415ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f0428c3d640 1 -- 172.21.15.107:0/2714767467 >> v1:172.21.15.195:6789/0 conn(0x7f04241588e0 legacy=0x7f04240c66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f0428c3d640 1 -- 172.21.15.107:0/2714767467 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042406dfa0 con 0x7f04240c8c10 2023-12-04T05:47:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f0428c3d640 1 -- 172.21.15.107:0/2714767467 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f0414001680 con 0x7f04240c8c10 2023-12-04T05:47:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f0428c3d640 1 -- 172.21.15.107:0/2714767467 >> v1:172.21.15.107:6789/0 conn(0x7f04240c8c10 legacy=0x7f04241581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f0428c3d640 1 --2- 172.21.15.107:0/2714767467 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0410003980 0x7f0410003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f0428c3d640 1 --2- 172.21.15.107:0/2714767467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100047d0 0x7f0410004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f0428c3d640 1 --2- 172.21.15.107:0/2714767467 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0410005190 0x7f041002d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f0428c3d640 1 -- 172.21.15.107:0/2714767467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04240c6290 con 0x7f04100047d0 2023-12-04T05:47:46.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f0428c3d640 1 -- 172.21.15.107:0/2714767467 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041002e050 con 0x7f0410005190 2023-12-04T05:47:46.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f0428c3d640 1 -- 172.21.15.107:0/2714767467 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f041002e210 con 0x7f0410003980 2023-12-04T05:47:46.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f042943e640 1 --2- 172.21.15.107:0/2714767467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100047d0 0x7f0410004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:46.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f042a440640 1 --2- 172.21.15.107:0/2714767467 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0410005190 0x7f041002d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:46.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f0429c3f640 1 --2- 172.21.15.107:0/2714767467 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0410003980 0x7f0410003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:46.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f042943e640 1 -- 172.21.15.107:0/2714767467 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0410003980 msgr2=0x7f0410003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:46.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f042943e640 1 --2- 172.21.15.107:0/2714767467 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f0410003980 0x7f0410003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:46.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f042943e640 1 -- 172.21.15.107:0/2714767467 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0410005190 msgr2=0x7f041002d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:46.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f042943e640 1 --2- 172.21.15.107:0/2714767467 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0410005190 0x7f041002d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:46.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f042943e640 1 -- 172.21.15.107:0/2714767467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0410003530 con 0x7f04100047d0 2023-12-04T05:47:46.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.393+0000 7f042a440640 1 -- 172.21.15.107:0/2714767467 reap_dead start 2023-12-04T05:47:46.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f042943e640 1 --2- 172.21.15.107:0/2714767467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100047d0 0x7f0410004c50 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0414001060 tx=0x7f041401cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7d1e5460b02e892 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:46.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f0428c3d640 1 -- 172.21.15.107:0/2714767467 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04140a5920 con 0x7f04100047d0 2023-12-04T05:47:46.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f0428c3d640 1 -- 172.21.15.107:0/2714767467 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04140a5a80 con 0x7f04100047d0 2023-12-04T05:47:46.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f0428c3d640 1 -- 172.21.15.107:0/2714767467 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04140060d0 con 0x7f04100047d0 2023-12-04T05:47:46.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f042bea3640 1 -- 172.21.15.107:0/2714767467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100047d0 msgr2=0x7f0410004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:46.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f042bea3640 1 --2- 172.21.15.107:0/2714767467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100047d0 0x7f0410004c50 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0414001060 tx=0x7f041401cc20 comp rx=0 tx=0).stop 2023-12-04T05:47:46.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f042bea3640 1 -- 172.21.15.107:0/2714767467 shutdown_connections 2023-12-04T05:47:46.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f042bea3640 1 --2- 172.21.15.107:0/2714767467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100047d0 0x7f0410004c50 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:46.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f042bea3640 1 -- 172.21.15.107:0/2714767467 >> 172.21.15.107:0/2714767467 conn(0x7f04240c7ff0 msgr2=0x7f04240c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:46.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f042bea3640 1 -- 172.21.15.107:0/2714767467 shutdown_connections 2023-12-04T05:47:46.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f042bea3640 1 -- 172.21.15.107:0/2714767467 wait complete. 2023-12-04T05:47:46.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f042bea3640 1 Processor -- start 2023-12-04T05:47:46.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f042bea3640 1 -- start start 2023-12-04T05:47:46.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f042bea3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100047d0 0x7f042413c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:46.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f042bea3640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0424159760 0x7f042415c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:46.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f042bea3640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f042413cd30 0x7f042415c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:46.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f042bea3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042413e2e0 con 0x7f04100047d0 2023-12-04T05:47:46.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f042bea3640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0424159e80 con 0x7f0424159760 2023-12-04T05:47:46.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f042bea3640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f042415db90 con 0x7f042413cd30 2023-12-04T05:47:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f0429c3f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100047d0 0x7f042413c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f0429c3f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100047d0 0x7f042413c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38218/0 (socket says 172.21.15.107:38218) 2023-12-04T05:47:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f0429c3f640 1 -- 172.21.15.107:0/2690903100 learned_addr learned my addr 172.21.15.107:0/2690903100 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f042943e640 1 --2- 172.21.15.107:0/2690903100 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0424159760 0x7f042415c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.397+0000 7f042a440640 1 --2- 172.21.15.107:0/2690903100 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f042413cd30 0x7f042415c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.401+0000 7f0429c3f640 1 -- 172.21.15.107:0/2690903100 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f042413cd30 msgr2=0x7f042415c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.401+0000 7f0429c3f640 1 --2- 172.21.15.107:0/2690903100 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f042413cd30 0x7f042415c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.401+0000 7f0429c3f640 1 -- 172.21.15.107:0/2690903100 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0424159760 msgr2=0x7f042415c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.401+0000 7f0429c3f640 1 --2- 172.21.15.107:0/2690903100 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0424159760 0x7f042415c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.401+0000 7f0429c3f640 1 -- 172.21.15.107:0/2690903100 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042415e270 con 0x7f04100047d0 2023-12-04T05:47:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.401+0000 7f0429c3f640 1 --2- 172.21.15.107:0/2690903100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100047d0 0x7f042413c7f0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f0420007f30 tx=0x7f0420011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.401+0000 7f041a7fc640 1 -- 172.21.15.107:0/2690903100 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042000e030 con 0x7f04100047d0 2023-12-04T05:47:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.401+0000 7f041a7fc640 1 -- 172.21.15.107:0/2690903100 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0420016470 con 0x7f04100047d0 2023-12-04T05:47:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.401+0000 7f042bea3640 1 -- 172.21.15.107:0/2690903100 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0424155c50 con 0x7f04100047d0 2023-12-04T05:47:46.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.401+0000 7f041a7fc640 1 -- 172.21.15.107:0/2690903100 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0420016820 con 0x7f04100047d0 2023-12-04T05:47:46.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.401+0000 7f042bea3640 1 -- 172.21.15.107:0/2690903100 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f042413d660 con 0x7f04100047d0 2023-12-04T05:47:46.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.401+0000 7f042bea3640 1 -- 172.21.15.107:0/2690903100 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0424067870 con 0x7f04100047d0 2023-12-04T05:47:46.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.401+0000 7f041a7fc640 1 -- 172.21.15.107:0/2690903100 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f0420016980 con 0x7f04100047d0 2023-12-04T05:47:46.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.401+0000 7f041a7fc640 1 --2- 172.21.15.107:0/2690903100 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0408066900 0x7f0408068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:46.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.401+0000 7f042943e640 1 --2- 172.21.15.107:0/2690903100 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0408066900 0x7f0408068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:46.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.401+0000 7f041a7fc640 1 -- 172.21.15.107:0/2690903100 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f042000f070 con 0x7f04100047d0 2023-12-04T05:47:46.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.401+0000 7f042943e640 1 --2- 172.21.15.107:0/2690903100 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0408066900 0x7f0408068dc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f04140aa000 tx=0x7f0414001a10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:46.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.405+0000 7f041a7fc640 1 -- 172.21.15.107:0/2690903100 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0420052660 con 0x7f04100047d0 2023-12-04T05:47:46.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.597+0000 7f042bea3640 1 -- 172.21.15.107:0/2690903100 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0424148940 con 0x7f04100047d0 2023-12-04T05:47:46.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.597+0000 7f041a7fc640 1 -- 172.21.15.107:0/2690903100 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f0420055c80 con 0x7f04100047d0 2023-12-04T05:47:46.602 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:46.602 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:46.602 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:46.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.601+0000 7f042bea3640 1 -- 172.21.15.107:0/2690903100 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0408066900 msgr2=0x7f0408068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:46.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.601+0000 7f042bea3640 1 --2- 172.21.15.107:0/2690903100 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0408066900 0x7f0408068dc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f04140aa000 tx=0x7f0414001a10 comp rx=0 tx=0).stop 2023-12-04T05:47:46.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.601+0000 7f042bea3640 1 -- 172.21.15.107:0/2690903100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100047d0 msgr2=0x7f042413c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:46.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.601+0000 7f042bea3640 1 --2- 172.21.15.107:0/2690903100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100047d0 0x7f042413c7f0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f0420007f30 tx=0x7f0420011660 comp rx=0 tx=0).stop 2023-12-04T05:47:46.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.601+0000 7f042bea3640 1 -- 172.21.15.107:0/2690903100 shutdown_connections 2023-12-04T05:47:46.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.601+0000 7f042bea3640 1 --2- 172.21.15.107:0/2690903100 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f0408066900 0x7f0408068dc0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:46.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.601+0000 7f042bea3640 1 --2- 172.21.15.107:0/2690903100 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f042413cd30 0x7f042415c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:46.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.601+0000 7f042bea3640 1 --2- 172.21.15.107:0/2690903100 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0424159760 0x7f042415c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:46.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.601+0000 7f042bea3640 1 --2- 172.21.15.107:0/2690903100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04100047d0 0x7f042413c7f0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:46.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.601+0000 7f042bea3640 1 -- 172.21.15.107:0/2690903100 >> 172.21.15.107:0/2690903100 conn(0x7f04240c7ff0 msgr2=0x7f042406d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:46.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.601+0000 7f042bea3640 1 -- 172.21.15.107:0/2690903100 shutdown_connections 2023-12-04T05:47:46.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:46.601+0000 7f042bea3640 1 -- 172.21.15.107:0/2690903100 wait complete. 2023-12-04T05:47:46.621 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:47:47.401 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~126s 2023-12-04T05:47:47.401 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~240s 2023-12-04T05:47:47.402 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~281s 2023-12-04T05:47:48.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:48.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.749+0000 7f918eb40640 1 Processor -- start 2023-12-04T05:47:48.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.749+0000 7f918eb40640 1 -- start start 2023-12-04T05:47:48.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f918eb40640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9188154290 0x7f918806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:48.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f918eb40640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f91880c8c80 0x7f91880c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:48.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f918eb40640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f918806e320 con 0x7f918815a000 2023-12-04T05:47:48.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f918eb40640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9188071c10 con 0x7f91880c8c80 2023-12-04T05:47:48.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f918eb40640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91880c66e0 con 0x7f9188154290 2023-12-04T05:47:48.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f9187fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f91880c8c80 0x7f91880c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:48.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f9187fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f91880c8c80 0x7f91880c6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49476/0 (socket says 172.21.15.107:49476) 2023-12-04T05:47:48.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f9187fff640 1 -- 172.21.15.107:0/3175697444 learned_addr learned my addr 172.21.15.107:0/3175697444 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:48.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f918c8dc640 1 --2- 172.21.15.107:0/3175697444 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9188154290 0x7f918806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:48.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f9187fff640 1 -- 172.21.15.107:0/3175697444 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9188154290 msgr2=0x7f918806fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:48.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f9187fff640 1 --2- 172.21.15.107:0/3175697444 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9188154290 0x7f918806fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:48.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f9187fff640 1 -- 172.21.15.107:0/3175697444 >> v1:172.21.15.195:6789/0 conn(0x7f918815a000 legacy=0x7f918815a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:48.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f9187fff640 1 -- 172.21.15.107:0/3175697444 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918806dd60 con 0x7f91880c8c80 2023-12-04T05:47:48.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f9187fff640 1 --2- 172.21.15.107:0/3175697444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91880c8c80 0x7f91880c6b70 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f917c009870 tx=0x7f917c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d2e8f6c8302ffc server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:48.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f91877fe640 1 -- 172.21.15.107:0/3175697444 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c0ab600 con 0x7f91880c8c80 2023-12-04T05:47:48.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f91877fe640 1 -- 172.21.15.107:0/3175697444 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917c0a7050 con 0x7f91880c8c80 2023-12-04T05:47:48.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f91877fe640 1 -- 172.21.15.107:0/3175697444 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c0ab980 con 0x7f91880c8c80 2023-12-04T05:47:48.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f918eb40640 1 -- 172.21.15.107:0/3175697444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91880c8c80 msgr2=0x7f91880c6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:48.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f918eb40640 1 --2- 172.21.15.107:0/3175697444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91880c8c80 0x7f91880c6b70 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f917c009870 tx=0x7f917c01c3e0 comp rx=0 tx=0).stop 2023-12-04T05:47:48.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f918eb40640 1 -- 172.21.15.107:0/3175697444 shutdown_connections 2023-12-04T05:47:48.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f918eb40640 1 --2- 172.21.15.107:0/3175697444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f91880c8c80 0x7f91880c6b70 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:48.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f918eb40640 1 --2- 172.21.15.107:0/3175697444 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9188154290 0x7f918806fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:48.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f918eb40640 1 -- 172.21.15.107:0/3175697444 >> 172.21.15.107:0/3175697444 conn(0x7f91880c7ff0 msgr2=0x7f91880c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:48.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.753+0000 7f918eb40640 1 -- 172.21.15.107:0/3175697444 shutdown_connections 2023-12-04T05:47:48.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f918eb40640 1 -- 172.21.15.107:0/3175697444 wait complete. 2023-12-04T05:47:48.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f918eb40640 1 Processor -- start 2023-12-04T05:47:48.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f918eb40640 1 -- start start 2023-12-04T05:47:48.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f918eb40640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f91880c8c80 0x7f9188142d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:48.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f918eb40640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9188154290 0x7f91880c34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:48.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f918eb40640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f918815a000 0x7f91880c3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:48.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f918eb40640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918813fc90 con 0x7f9188154290 2023-12-04T05:47:48.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f918eb40640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918814d7f0 con 0x7f918815a000 2023-12-04T05:47:48.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f918eb40640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f918813e7e0 con 0x7f91880c8c80 2023-12-04T05:47:48.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f9187fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9188154290 0x7f91880c34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:48.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f9187fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9188154290 0x7f91880c34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49478/0 (socket says 172.21.15.107:49478) 2023-12-04T05:47:48.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f9187fff640 1 -- 172.21.15.107:0/2073820709 learned_addr learned my addr 172.21.15.107:0/2073820709 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:48.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f918c8dc640 1 --2- 172.21.15.107:0/2073820709 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f91880c8c80 0x7f9188142d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:48.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f918d0dd640 1 --2- 172.21.15.107:0/2073820709 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f918815a000 0x7f91880c3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:48.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f9187fff640 1 -- 172.21.15.107:0/2073820709 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f91880c8c80 msgr2=0x7f9188142d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:48.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f9187fff640 1 --2- 172.21.15.107:0/2073820709 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f91880c8c80 0x7f9188142d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:48.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f9187fff640 1 -- 172.21.15.107:0/2073820709 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f918815a000 msgr2=0x7f91880c3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:48.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f9187fff640 1 --2- 172.21.15.107:0/2073820709 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f918815a000 0x7f91880c3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:48.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f9187fff640 1 -- 172.21.15.107:0/2073820709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f917c0094d0 con 0x7f9188154290 2023-12-04T05:47:48.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f918d0dd640 1 --2- 172.21.15.107:0/2073820709 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f918815a000 0x7f91880c3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:47:48.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f9187fff640 1 --2- 172.21.15.107:0/2073820709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9188154290 0x7f91880c34d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f917c0ae040 tx=0x7f917c01ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:48.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f91857fa640 1 -- 172.21.15.107:0/2073820709 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c0b6620 con 0x7f9188154290 2023-12-04T05:47:48.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f91857fa640 1 -- 172.21.15.107:0/2073820709 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917c0abe90 con 0x7f9188154290 2023-12-04T05:47:48.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f918eb40640 1 -- 172.21.15.107:0/2073820709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f918814da40 con 0x7f9188154290 2023-12-04T05:47:48.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f91857fa640 1 -- 172.21.15.107:0/2073820709 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c0b69e0 con 0x7f9188154290 2023-12-04T05:47:48.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f918eb40640 1 -- 172.21.15.107:0/2073820709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f918814e000 con 0x7f9188154290 2023-12-04T05:47:48.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.757+0000 7f918eb40640 1 -- 172.21.15.107:0/2073820709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91880c31f0 con 0x7f9188154290 2023-12-04T05:47:48.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.761+0000 7f91857fa640 1 -- 172.21.15.107:0/2073820709 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f917c0b2040 con 0x7f9188154290 2023-12-04T05:47:48.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.761+0000 7f91857fa640 1 --2- 172.21.15.107:0/2073820709 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f91740668d0 0x7f9174068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:48.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.761+0000 7f91857fa640 1 -- 172.21.15.107:0/2073820709 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f917c11eb40 con 0x7f9188154290 2023-12-04T05:47:48.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.761+0000 7f918c8dc640 1 --2- 172.21.15.107:0/2073820709 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f91740668d0 0x7f9174068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:48.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.761+0000 7f918c8dc640 1 --2- 172.21.15.107:0/2073820709 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f91740668d0 0x7f9174068d90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f917800a970 tx=0x7f9178008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:48.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.765+0000 7f91857fa640 1 -- 172.21.15.107:0/2073820709 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f91880c31f0 con 0x7f9188154290 2023-12-04T05:47:48.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.945+0000 7f918eb40640 1 -- 172.21.15.107:0/2073820709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91880c31f0 con 0x7f9188154290 2023-12-04T05:47:48.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.949+0000 7f91857fa640 1 -- 172.21.15.107:0/2073820709 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f91880c31f0 con 0x7f9188154290 2023-12-04T05:47:48.950 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:48.950 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:48.951 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.949+0000 7f918eb40640 1 -- 172.21.15.107:0/2073820709 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f91740668d0 msgr2=0x7f9174068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.949+0000 7f918eb40640 1 --2- 172.21.15.107:0/2073820709 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f91740668d0 0x7f9174068d90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f917800a970 tx=0x7f9178008040 comp rx=0 tx=0).stop 2023-12-04T05:47:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.949+0000 7f918eb40640 1 -- 172.21.15.107:0/2073820709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9188154290 msgr2=0x7f91880c34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.949+0000 7f918eb40640 1 --2- 172.21.15.107:0/2073820709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9188154290 0x7f91880c34d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f917c0ae040 tx=0x7f917c01ac20 comp rx=0 tx=0).stop 2023-12-04T05:47:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.949+0000 7f918eb40640 1 -- 172.21.15.107:0/2073820709 shutdown_connections 2023-12-04T05:47:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.949+0000 7f918eb40640 1 --2- 172.21.15.107:0/2073820709 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f91740668d0 0x7f9174068d90 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.949+0000 7f918eb40640 1 --2- 172.21.15.107:0/2073820709 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f918815a000 0x7f91880c3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.949+0000 7f918eb40640 1 --2- 172.21.15.107:0/2073820709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9188154290 0x7f91880c34d0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.949+0000 7f918eb40640 1 --2- 172.21.15.107:0/2073820709 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f91880c8c80 0x7f9188142d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.949+0000 7f918eb40640 1 -- 172.21.15.107:0/2073820709 >> 172.21.15.107:0/2073820709 conn(0x7f91880c7ff0 msgr2=0x7f91880c6a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.949+0000 7f918eb40640 1 -- 172.21.15.107:0/2073820709 shutdown_connections 2023-12-04T05:47:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:48.949+0000 7f918eb40640 1 -- 172.21.15.107:0/2073820709 wait complete. 2023-12-04T05:47:48.967 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:47:50.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:51.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.093+0000 7f9d7758d640 1 Processor -- start 2023-12-04T05:47:51.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.093+0000 7f9d7758d640 1 -- start start 2023-12-04T05:47:51.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.097+0000 7f9d7758d640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9d70071c00 0x7f9d700c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:51.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.097+0000 7f9d7758d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d70070380 0x7f9d700707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:51.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.097+0000 7f9d7758d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d700700c0 con 0x7f9d70151180 2023-12-04T05:47:51.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.097+0000 7f9d7758d640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d70156df0 con 0x7f9d70071c00 2023-12-04T05:47:51.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.097+0000 7f9d7758d640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d701570d0 con 0x7f9d70070380 2023-12-04T05:47:51.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.097+0000 7f9d75b2a640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d70070380 0x7f9d700707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:51.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.097+0000 7f9d74b28640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9d70071c00 0x7f9d700c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:51.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.097+0000 7f9d74b28640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9d70071c00 0x7f9d700c79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.107:51342/0 (socket says 172.21.15.107:51342) 2023-12-04T05:47:51.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.097+0000 7f9d75b2a640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d70070380 0x7f9d700707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.107:46480/0 (socket says 172.21.15.107:46480) 2023-12-04T05:47:51.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.097+0000 7f9d74b28640 1 -- 172.21.15.107:0/1247782823 learned_addr learned my addr 172.21.15.107:0/1247782823 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:51.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.097+0000 7f9d74b28640 1 -- 172.21.15.107:0/1247782823 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d70070380 msgr2=0x7f9d700707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:51.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.097+0000 7f9d74b28640 1 --2- 172.21.15.107:0/1247782823 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d70070380 0x7f9d700707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:51.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.097+0000 7f9d74b28640 1 -- 172.21.15.107:0/1247782823 >> v1:172.21.15.195:6789/0 conn(0x7f9d70151180 legacy=0x7f9d700714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:51.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.097+0000 7f9d74b28640 1 -- 172.21.15.107:0/1247782823 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d701573b0 con 0x7f9d70071c00 2023-12-04T05:47:51.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.097+0000 7f9d75b2a640 1 --2- 172.21.15.107:0/1247782823 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d70070380 0x7f9d700707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T05:47:51.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.101+0000 7f9d74b28640 1 --2- 172.21.15.107:0/1247782823 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d70071c00 0x7f9d700c79e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9d60009870 tx=0x7f9d6001ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0623964bfc60850 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:51.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.101+0000 7f9d67fff640 1 -- 172.21.15.107:0/1247782823 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d600a8a30 con 0x7f9d70071c00 2023-12-04T05:47:51.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.101+0000 7f9d67fff640 1 -- 172.21.15.107:0/1247782823 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d600a8b90 con 0x7f9d70071c00 2023-12-04T05:47:51.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.101+0000 7f9d67fff640 1 -- 172.21.15.107:0/1247782823 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d600b0440 con 0x7f9d70071c00 2023-12-04T05:47:51.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.101+0000 7f9d7758d640 1 -- 172.21.15.107:0/1247782823 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d70071c00 msgr2=0x7f9d700c79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:51.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.101+0000 7f9d7758d640 1 --2- 172.21.15.107:0/1247782823 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d70071c00 0x7f9d700c79e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9d60009870 tx=0x7f9d6001ed30 comp rx=0 tx=0).stop 2023-12-04T05:47:51.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.101+0000 7f9d7758d640 1 -- 172.21.15.107:0/1247782823 shutdown_connections 2023-12-04T05:47:51.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.101+0000 7f9d7758d640 1 --2- 172.21.15.107:0/1247782823 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d70070380 0x7f9d700707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:51.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.101+0000 7f9d7758d640 1 --2- 172.21.15.107:0/1247782823 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d70071c00 0x7f9d700c79e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:51.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.101+0000 7f9d7758d640 1 -- 172.21.15.107:0/1247782823 >> 172.21.15.107:0/1247782823 conn(0x7f9d700c7ff0 msgr2=0x7f9d700c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:51.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.101+0000 7f9d7758d640 1 -- 172.21.15.107:0/1247782823 shutdown_connections 2023-12-04T05:47:51.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.101+0000 7f9d7758d640 1 -- 172.21.15.107:0/1247782823 wait complete. 2023-12-04T05:47:51.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.101+0000 7f9d7758d640 1 Processor -- start 2023-12-04T05:47:51.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.101+0000 7f9d7758d640 1 -- start start 2023-12-04T05:47:51.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d7758d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d70070380 0x7f9d70154970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:51.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d7758d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d70071c00 0x7f9d701f0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:51.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d7758d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d70151180 0x7f9d701f05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:51.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d7758d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d701557f0 con 0x7f9d70070380 2023-12-04T05:47:51.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d7758d640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d700c56f0 con 0x7f9d70151180 2023-12-04T05:47:51.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d7758d640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d701e2d80 con 0x7f9d70071c00 2023-12-04T05:47:51.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d75329640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d70070380 0x7f9d70154970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:51.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d75329640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d70070380 0x7f9d70154970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49482/0 (socket says 172.21.15.107:49482) 2023-12-04T05:47:51.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d75329640 1 -- 172.21.15.107:0/3630717479 learned_addr learned my addr 172.21.15.107:0/3630717479 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:51.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d74b28640 1 --2- 172.21.15.107:0/3630717479 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d70071c00 0x7f9d701f0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:51.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d75b2a640 1 --2- 172.21.15.107:0/3630717479 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d70151180 0x7f9d701f05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:51.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d75329640 1 -- 172.21.15.107:0/3630717479 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d70071c00 msgr2=0x7f9d701f0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:51.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d75329640 1 --2- 172.21.15.107:0/3630717479 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d70071c00 0x7f9d701f0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:51.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d75329640 1 -- 172.21.15.107:0/3630717479 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d70151180 msgr2=0x7f9d701f05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:51.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d75329640 1 --2- 172.21.15.107:0/3630717479 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d70151180 0x7f9d701f05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:51.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d75329640 1 -- 172.21.15.107:0/3630717479 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d68009a70 con 0x7f9d70070380 2023-12-04T05:47:51.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d75329640 1 --2- 172.21.15.107:0/3630717479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d70070380 0x7f9d70154970 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00c390 tx=0x7f9d6c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:51.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d65ffb640 1 -- 172.21.15.107:0/3630717479 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c0052d0 con 0x7f9d70070380 2023-12-04T05:47:51.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d65ffb640 1 -- 172.21.15.107:0/3630717479 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6c005450 con 0x7f9d70070380 2023-12-04T05:47:51.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d7758d640 1 -- 172.21.15.107:0/3630717479 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d600094d0 con 0x7f9d70070380 2023-12-04T05:47:51.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d65ffb640 1 -- 172.21.15.107:0/3630717479 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c003840 con 0x7f9d70070380 2023-12-04T05:47:51.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d7758d640 1 -- 172.21.15.107:0/3630717479 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d701e3040 con 0x7f9d70070380 2023-12-04T05:47:51.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d7758d640 1 -- 172.21.15.107:0/3630717479 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d700c5c20 con 0x7f9d70070380 2023-12-04T05:47:51.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d65ffb640 1 -- 172.21.15.107:0/3630717479 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9d6c0039a0 con 0x7f9d70070380 2023-12-04T05:47:51.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d65ffb640 1 --2- 172.21.15.107:0/3630717479 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9d54066900 0x7f9d54068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:51.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d65ffb640 1 -- 172.21.15.107:0/3630717479 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f9d6c083ad0 con 0x7f9d70070380 2023-12-04T05:47:51.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.105+0000 7f9d74b28640 1 --2- 172.21.15.107:0/3630717479 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9d54066900 0x7f9d54068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:51.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.109+0000 7f9d74b28640 1 --2- 172.21.15.107:0/3630717479 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9d54066900 0x7f9d54068dc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9d600b2000 tx=0x7f9d600aa070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:51.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.113+0000 7f9d65ffb640 1 -- 172.21.15.107:0/3630717479 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9d6c050b50 con 0x7f9d70070380 2023-12-04T05:47:51.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.293+0000 7f9d7758d640 1 -- 172.21.15.107:0/3630717479 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d70067870 con 0x7f9d70070380 2023-12-04T05:47:51.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.297+0000 7f9d65ffb640 1 -- 172.21.15.107:0/3630717479 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f9d6c054170 con 0x7f9d70070380 2023-12-04T05:47:51.299 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:51.299 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:51.299 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:51.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.297+0000 7f9d7758d640 1 -- 172.21.15.107:0/3630717479 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9d54066900 msgr2=0x7f9d54068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:51.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.297+0000 7f9d7758d640 1 --2- 172.21.15.107:0/3630717479 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9d54066900 0x7f9d54068dc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9d600b2000 tx=0x7f9d600aa070 comp rx=0 tx=0).stop 2023-12-04T05:47:51.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.297+0000 7f9d7758d640 1 -- 172.21.15.107:0/3630717479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d70070380 msgr2=0x7f9d70154970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:51.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.297+0000 7f9d7758d640 1 --2- 172.21.15.107:0/3630717479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d70070380 0x7f9d70154970 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00c390 tx=0x7f9d6c00c850 comp rx=0 tx=0).stop 2023-12-04T05:47:51.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.297+0000 7f9d7758d640 1 -- 172.21.15.107:0/3630717479 shutdown_connections 2023-12-04T05:47:51.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.297+0000 7f9d7758d640 1 --2- 172.21.15.107:0/3630717479 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9d54066900 0x7f9d54068dc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:51.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.297+0000 7f9d7758d640 1 --2- 172.21.15.107:0/3630717479 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9d70151180 0x7f9d701f05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:51.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.297+0000 7f9d7758d640 1 --2- 172.21.15.107:0/3630717479 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9d70071c00 0x7f9d701f0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:51.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.297+0000 7f9d7758d640 1 --2- 172.21.15.107:0/3630717479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d70070380 0x7f9d70154970 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:51.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.297+0000 7f9d7758d640 1 -- 172.21.15.107:0/3630717479 >> 172.21.15.107:0/3630717479 conn(0x7f9d700c7ff0 msgr2=0x7f9d700739f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:51.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.297+0000 7f9d7758d640 1 -- 172.21.15.107:0/3630717479 shutdown_connections 2023-12-04T05:47:51.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:51.297+0000 7f9d7758d640 1 -- 172.21.15.107:0/3630717479 wait complete. 2023-12-04T05:47:51.317 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 50, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-12-04T05:39:59.032019+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_4288": { "addr": "172.21.15.195:6837/3412865477", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6835", "nonce": 3412865477, "type": "v2" }, { "addr": "172.21.15.195:6837", "nonce": 3412865477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_5190": { "addr": "172.21.15.107:6836/2928623624", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 2928623624, "type": "v2" }, { "addr": "172.21.15.107:6836", "nonce": 2928623624, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5190, "incarnation": 45, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:rejoin", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-12-04T05:46:11.040247+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2, 3, 4 ], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 5190 } } } ], "standbys": [ { "addr": "172.21.15.107:6837/2752497349", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6835", "nonce": 2752497349, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 2752497349, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 49, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5292, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-12-04T05:47:53.318 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:53.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.445+0000 7fe6a246d640 1 Processor -- start 2023-12-04T05:47:53.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.445+0000 7fe6a246d640 1 -- start start 2023-12-04T05:47:53.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe6a246d640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe69c0d01c0 0x7fe69c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:53.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe6a246d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe69c072220 0x7fe69c0c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:53.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe6a246d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe69c15d710 con 0x7fe69c154020 2023-12-04T05:47:53.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe6a246d640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe69c0ca230 con 0x7fe69c0d01c0 2023-12-04T05:47:53.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe6a246d640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe69c0c6770 con 0x7fe69c072220 2023-12-04T05:47:53.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe69bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe69c154020 0x7fe69c15a810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48086/0 (socket says 172.21.15.107:48086) 2023-12-04T05:47:53.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe69bfff640 1 -- 172.21.15.107:0/3935049305 learned_addr learned my addr 172.21.15.107:0/3935049305 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:47:53.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe69affd640 1 -- 172.21.15.107:0/3935049305 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321886202 0 0) 0x7fe69c15d710 con 0x7fe69c154020 2023-12-04T05:47:53.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe69affd640 1 -- 172.21.15.107:0/3935049305 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe684003530 con 0x7fe69c154020 2023-12-04T05:47:53.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe6a0a0a640 1 --2- 172.21.15.107:0/3935049305 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe69c072220 0x7fe69c0c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:53.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe69b7fe640 1 --2- 172.21.15.107:0/3935049305 >> v2:172.21.15.195:3300/0 conn(0x7fe69c0d01c0 0x7fe69c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:53.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe69affd640 1 -- 172.21.15.107:0/3935049305 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 166983316 0 0) 0x7fe684003530 con 0x7fe69c154020 2023-12-04T05:47:53.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe69affd640 1 -- 172.21.15.107:0/3935049305 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe69c072220 msgr2=0x7fe69c0c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:53.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe69affd640 1 --2- 172.21.15.107:0/3935049305 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe69c072220 0x7fe69c0c4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:53.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe69affd640 1 -- 172.21.15.107:0/3935049305 >> v2:172.21.15.195:3300/0 conn(0x7fe69c0d01c0 msgr2=0x7fe69c071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:53.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe69affd640 1 --2- 172.21.15.107:0/3935049305 >> v2:172.21.15.195:3300/0 conn(0x7fe69c0d01c0 0x7fe69c071ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:53.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe69affd640 1 -- 172.21.15.107:0/3935049305 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69c15d9f0 con 0x7fe69c154020 2023-12-04T05:47:53.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe69affd640 1 -- 172.21.15.107:0/3935049305 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7fe68c0016a0 con 0x7fe69c154020 2023-12-04T05:47:53.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe69affd640 1 -- 172.21.15.107:0/3935049305 >> v1:172.21.15.107:6789/0 conn(0x7fe69c154020 legacy=0x7fe69c15a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:53.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe69affd640 1 --2- 172.21.15.107:0/3935049305 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6840039b0 0x7fe684003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:53.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe69affd640 1 --2- 172.21.15.107:0/3935049305 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe69c072220 0x7fe6840047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:53.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe69affd640 1 --2- 172.21.15.107:0/3935049305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe684004d10 0x7fe684005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:53.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe69affd640 1 -- 172.21.15.107:0/3935049305 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68400bad0 con 0x7fe684004d10 2023-12-04T05:47:53.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe69affd640 1 -- 172.21.15.107:0/3935049305 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68400bc10 con 0x7fe6840039b0 2023-12-04T05:47:53.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe69affd640 1 -- 172.21.15.107:0/3935049305 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe68400bd80 con 0x7fe69c072220 2023-12-04T05:47:53.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe6a0a0a640 1 --2- 172.21.15.107:0/3935049305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe684004d10 0x7fe684005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:53.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe69bfff640 1 --2- 172.21.15.107:0/3935049305 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6840039b0 0x7fe684003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:53.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe6a0a0a640 1 -- 172.21.15.107:0/3935049305 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe69c072220 msgr2=0x7fe6840047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:53.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe6a0a0a640 1 --2- 172.21.15.107:0/3935049305 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe69c072220 0x7fe6840047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:53.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe6a0a0a640 1 -- 172.21.15.107:0/3935049305 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6840039b0 msgr2=0x7fe684003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:53.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe6a0a0a640 1 --2- 172.21.15.107:0/3935049305 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6840039b0 0x7fe684003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:53.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe6a0a0a640 1 -- 172.21.15.107:0/3935049305 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe684003530 con 0x7fe684004d10 2023-12-04T05:47:53.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.449+0000 7fe69bfff640 1 --2- 172.21.15.107:0/3935049305 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe6840039b0 0x7fe684003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:47:53.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe6a0a0a640 1 --2- 172.21.15.107:0/3935049305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe684004d10 0x7fe684005190 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe694007360 tx=0x7fe69401ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b2a99f7671ae808 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:53.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe69affd640 1 -- 172.21.15.107:0/3935049305 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6940a5d60 con 0x7fe684004d10 2023-12-04T05:47:53.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe69affd640 1 -- 172.21.15.107:0/3935049305 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6940ad440 con 0x7fe684004d10 2023-12-04T05:47:53.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe69affd640 1 -- 172.21.15.107:0/3935049305 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6940ad7d0 con 0x7fe684004d10 2023-12-04T05:47:53.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe6a246d640 1 -- 172.21.15.107:0/3935049305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe684004d10 msgr2=0x7fe684005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:53.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe6a246d640 1 --2- 172.21.15.107:0/3935049305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe684004d10 0x7fe684005190 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe694007360 tx=0x7fe69401ce00 comp rx=0 tx=0).stop 2023-12-04T05:47:53.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe6a0a0a640 1 -- 172.21.15.107:0/3935049305 reap_dead start 2023-12-04T05:47:53.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe6a246d640 1 -- 172.21.15.107:0/3935049305 shutdown_connections 2023-12-04T05:47:53.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe6a246d640 1 -- 172.21.15.107:0/3935049305 >> 172.21.15.107:0/3935049305 conn(0x7fe69c06c6c0 msgr2=0x7fe69c0cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:53.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe6a246d640 1 -- 172.21.15.107:0/3935049305 shutdown_connections 2023-12-04T05:47:53.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe6a246d640 1 -- 172.21.15.107:0/3935049305 wait complete. 2023-12-04T05:47:53.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe6a246d640 1 Processor -- start 2023-12-04T05:47:53.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe6a246d640 1 -- start start 2023-12-04T05:47:53.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe6a246d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe69c1f27a0 0x7fe69c0b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:53.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe6a246d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe69c15af50 0x7fe69c1f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:53.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe6a246d640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe69c1f30c0 0x7fe69c145b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:53.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe6a246d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69c0b3950 con 0x7fe69c1f27a0 2023-12-04T05:47:53.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe6a246d640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69c15d620 con 0x7fe69c15af50 2023-12-04T05:47:53.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe6a246d640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe69c0b4320 con 0x7fe69c1f30c0 2023-12-04T05:47:53.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe69bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe69c1f27a0 0x7fe69c0b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:53.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe69bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe69c1f27a0 0x7fe69c0b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49498/0 (socket says 172.21.15.107:49498) 2023-12-04T05:47:53.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe69bfff640 1 -- 172.21.15.107:0/1864405413 learned_addr learned my addr 172.21.15.107:0/1864405413 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:53.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe69b7fe640 1 --2- 172.21.15.107:0/1864405413 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe69c15af50 0x7fe69c1f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:53.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe6a0a0a640 1 --2- 172.21.15.107:0/1864405413 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe69c1f30c0 0x7fe69c145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe69bfff640 1 -- 172.21.15.107:0/1864405413 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe69c1f30c0 msgr2=0x7fe69c145b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe69bfff640 1 --2- 172.21.15.107:0/1864405413 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe69c1f30c0 0x7fe69c145b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe69bfff640 1 -- 172.21.15.107:0/1864405413 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe69c15af50 msgr2=0x7fe69c1f2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe69bfff640 1 --2- 172.21.15.107:0/1864405413 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe69c15af50 0x7fe69c1f2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe69bfff640 1 -- 172.21.15.107:0/1864405413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69c159500 con 0x7fe69c1f27a0 2023-12-04T05:47:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe69bfff640 1 --2- 172.21.15.107:0/1864405413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe69c1f27a0 0x7fe69c0b3de0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe68c000f70 tx=0x7fe68c00df70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe698ff9640 1 -- 172.21.15.107:0/1864405413 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68c004cc0 con 0x7fe69c1f27a0 2023-12-04T05:47:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe698ff9640 1 -- 172.21.15.107:0/1864405413 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe68c004e20 con 0x7fe69c1f27a0 2023-12-04T05:47:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe6a246d640 1 -- 172.21.15.107:0/1864405413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe69c1597e0 con 0x7fe69c1f27a0 2023-12-04T05:47:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe698ff9640 1 -- 172.21.15.107:0/1864405413 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68c0072d0 con 0x7fe69c1f27a0 2023-12-04T05:47:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.453+0000 7fe6a246d640 1 -- 172.21.15.107:0/1864405413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe69c0b47c0 con 0x7fe69c1f27a0 2023-12-04T05:47:53.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.457+0000 7fe6a246d640 1 -- 172.21.15.107:0/1864405413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe69c0b3440 con 0x7fe69c1f27a0 2023-12-04T05:47:53.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.457+0000 7fe698ff9640 1 -- 172.21.15.107:0/1864405413 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe68c007430 con 0x7fe69c1f27a0 2023-12-04T05:47:53.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.457+0000 7fe698ff9640 1 --2- 172.21.15.107:0/1864405413 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe688066900 0x7fe688068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:53.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.457+0000 7fe698ff9640 1 -- 172.21.15.107:0/1864405413 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fe68c084340 con 0x7fe69c1f27a0 2023-12-04T05:47:53.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.457+0000 7fe69b7fe640 1 --2- 172.21.15.107:0/1864405413 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe688066900 0x7fe688068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:53.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.457+0000 7fe69b7fe640 1 --2- 172.21.15.107:0/1864405413 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe688066900 0x7fe688068dc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe69c0c9b90 tx=0x7fe690008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:53.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.461+0000 7fe698ff9640 1 -- 172.21.15.107:0/1864405413 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe69c0b3440 con 0x7fe69c1f27a0 2023-12-04T05:47:53.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.633+0000 7fe6a246d640 1 -- 172.21.15.107:0/1864405413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe69c0b3440 con 0x7fe69c1f27a0 2023-12-04T05:47:53.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.633+0000 7fe698ff9640 1 -- 172.21.15.107:0/1864405413 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7fe69c0b3440 con 0x7fe69c1f27a0 2023-12-04T05:47:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:53.637 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:53.637 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:53.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.637+0000 7fe6a246d640 1 -- 172.21.15.107:0/1864405413 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe688066900 msgr2=0x7fe688068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:53.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.637+0000 7fe6a246d640 1 --2- 172.21.15.107:0/1864405413 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe688066900 0x7fe688068dc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe69c0c9b90 tx=0x7fe690008040 comp rx=0 tx=0).stop 2023-12-04T05:47:53.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.637+0000 7fe6a246d640 1 -- 172.21.15.107:0/1864405413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe69c1f27a0 msgr2=0x7fe69c0b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:53.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.637+0000 7fe6a246d640 1 --2- 172.21.15.107:0/1864405413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe69c1f27a0 0x7fe69c0b3de0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe68c000f70 tx=0x7fe68c00df70 comp rx=0 tx=0).stop 2023-12-04T05:47:53.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.637+0000 7fe6a246d640 1 -- 172.21.15.107:0/1864405413 shutdown_connections 2023-12-04T05:47:53.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.637+0000 7fe6a246d640 1 --2- 172.21.15.107:0/1864405413 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe688066900 0x7fe688068dc0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:53.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.637+0000 7fe6a246d640 1 --2- 172.21.15.107:0/1864405413 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe69c1f30c0 0x7fe69c145b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:53.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.637+0000 7fe6a246d640 1 --2- 172.21.15.107:0/1864405413 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe69c15af50 0x7fe69c1f2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:53.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.637+0000 7fe6a246d640 1 --2- 172.21.15.107:0/1864405413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe69c1f27a0 0x7fe69c0b3de0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:53.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.637+0000 7fe6a246d640 1 -- 172.21.15.107:0/1864405413 >> 172.21.15.107:0/1864405413 conn(0x7fe69c06c6c0 msgr2=0x7fe69c0ca780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:53.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.637+0000 7fe6a246d640 1 -- 172.21.15.107:0/1864405413 shutdown_connections 2023-12-04T05:47:53.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:53.637+0000 7fe6a246d640 1 -- 172.21.15.107:0/1864405413 wait complete. 2023-12-04T05:47:53.655 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:47:54.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~132s 2023-12-04T05:47:54.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~246s 2023-12-04T05:47:54.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~287s 2023-12-04T05:47:55.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:55.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.781+0000 7f790c0dc640 1 Processor -- start 2023-12-04T05:47:55.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.781+0000 7f790c0dc640 1 -- start start 2023-12-04T05:47:55.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.785+0000 7f790c0dc640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f790406b6c0 0x7f790406baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:55.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.785+0000 7f790c0dc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7904072430 0x7f790415b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:55.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.785+0000 7f790c0dc640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790415ac80 con 0x7f790415bdd0 2023-12-04T05:47:55.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.785+0000 7f790c0dc640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f790406bfe0 con 0x7f7904072430 2023-12-04T05:47:55.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.785+0000 7f790c0dc640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f790415c260 con 0x7f790406b6c0 2023-12-04T05:47:55.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.785+0000 7f7909677640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7904072430 0x7f790415b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:55.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.785+0000 7f7909677640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7904072430 0x7f790415b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49510/0 (socket says 172.21.15.107:49510) 2023-12-04T05:47:55.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.785+0000 7f7909677640 1 -- 172.21.15.107:0/4201755924 learned_addr learned my addr 172.21.15.107:0/4201755924 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:55.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.785+0000 7f7909e78640 1 --2- 172.21.15.107:0/4201755924 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f790406b6c0 0x7f790406baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:55.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.785+0000 7f7909677640 1 -- 172.21.15.107:0/4201755924 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f790406b6c0 msgr2=0x7f790406baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:55.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.785+0000 7f7909677640 1 --2- 172.21.15.107:0/4201755924 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f790406b6c0 0x7f790406baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:55.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.785+0000 7f7909677640 1 -- 172.21.15.107:0/4201755924 >> v1:172.21.15.195:6789/0 conn(0x7f790415bdd0 legacy=0x7f790415d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:55.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.785+0000 7f7909677640 1 -- 172.21.15.107:0/4201755924 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790415a100 con 0x7f7904072430 2023-12-04T05:47:55.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.785+0000 7f7909677640 1 --2- 172.21.15.107:0/4201755924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7904072430 0x7f790415b890 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f78f8009870 tx=0x7f78f801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b24927e2fcec0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:55.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.785+0000 7f7908e76640 1 -- 172.21.15.107:0/4201755924 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f80ab600 con 0x7f7904072430 2023-12-04T05:47:55.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f7908e76640 1 -- 172.21.15.107:0/4201755924 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f80a7050 con 0x7f7904072430 2023-12-04T05:47:55.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f7908e76640 1 -- 172.21.15.107:0/4201755924 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f80ab980 con 0x7f7904072430 2023-12-04T05:47:55.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f790c0dc640 1 -- 172.21.15.107:0/4201755924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7904072430 msgr2=0x7f790415b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:55.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f790c0dc640 1 --2- 172.21.15.107:0/4201755924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7904072430 0x7f790415b890 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f78f8009870 tx=0x7f78f801c500 comp rx=0 tx=0).stop 2023-12-04T05:47:55.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f790c0dc640 1 -- 172.21.15.107:0/4201755924 shutdown_connections 2023-12-04T05:47:55.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f790c0dc640 1 --2- 172.21.15.107:0/4201755924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7904072430 0x7f790415b890 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:55.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f790c0dc640 1 --2- 172.21.15.107:0/4201755924 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f790406b6c0 0x7f790406baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:55.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f790c0dc640 1 -- 172.21.15.107:0/4201755924 >> 172.21.15.107:0/4201755924 conn(0x7f79040663a0 msgr2=0x7f79040667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:55.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f790c0dc640 1 -- 172.21.15.107:0/4201755924 shutdown_connections 2023-12-04T05:47:55.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f790c0dc640 1 -- 172.21.15.107:0/4201755924 wait complete. 2023-12-04T05:47:55.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f790c0dc640 1 Processor -- start 2023-12-04T05:47:55.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f790c0dc640 1 -- start start 2023-12-04T05:47:55.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f790c0dc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f790406b6c0 0x7f79040c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:55.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f790c0dc640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7904072430 0x7f7904148b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:55.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f790c0dc640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f790415bdd0 0x7f79040c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:55.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f790c0dc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7904158e60 con 0x7f790406b6c0 2023-12-04T05:47:55.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f790c0dc640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79041f3bc0 con 0x7f7904072430 2023-12-04T05:47:55.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f790c0dc640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79041ec400 con 0x7f790415bdd0 2023-12-04T05:47:55.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f7909e78640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f790406b6c0 0x7f79040c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:55.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f7909e78640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f790406b6c0 0x7f79040c8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49522/0 (socket says 172.21.15.107:49522) 2023-12-04T05:47:55.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f7909e78640 1 -- 172.21.15.107:0/965663895 learned_addr learned my addr 172.21.15.107:0/965663895 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:55.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f7909677640 1 --2- 172.21.15.107:0/965663895 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7904072430 0x7f7904148b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:55.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f790a679640 1 --2- 172.21.15.107:0/965663895 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f790415bdd0 0x7f79040c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:55.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f7909e78640 1 -- 172.21.15.107:0/965663895 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f790415bdd0 msgr2=0x7f79040c92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:55.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f7909e78640 1 --2- 172.21.15.107:0/965663895 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f790415bdd0 0x7f79040c92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:55.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f7909e78640 1 -- 172.21.15.107:0/965663895 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7904072430 msgr2=0x7f7904148b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:55.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f7909e78640 1 --2- 172.21.15.107:0/965663895 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7904072430 0x7f7904148b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:55.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f7909e78640 1 -- 172.21.15.107:0/965663895 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f80094d0 con 0x7f790406b6c0 2023-12-04T05:47:55.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f7909677640 1 --2- 172.21.15.107:0/965663895 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7904072430 0x7f7904148b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:47:55.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f7909e78640 1 --2- 172.21.15.107:0/965663895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f790406b6c0 0x7f79040c8bf0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f78f4007e40 tx=0x7f78f400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:55.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f79027fc640 1 -- 172.21.15.107:0/965663895 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f4014620 con 0x7f790406b6c0 2023-12-04T05:47:55.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f79027fc640 1 -- 172.21.15.107:0/965663895 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f400ee90 con 0x7f790406b6c0 2023-12-04T05:47:55.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f79027fc640 1 -- 172.21.15.107:0/965663895 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f40149c0 con 0x7f790406b6c0 2023-12-04T05:47:55.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f790c0dc640 1 -- 172.21.15.107:0/965663895 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79041ec6e0 con 0x7f790406b6c0 2023-12-04T05:47:55.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.789+0000 7f790c0dc640 1 -- 172.21.15.107:0/965663895 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79041ecca0 con 0x7f790406b6c0 2023-12-04T05:47:55.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.793+0000 7f790c0dc640 1 -- 172.21.15.107:0/965663895 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79040c4780 con 0x7f790406b6c0 2023-12-04T05:47:55.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.793+0000 7f79027fc640 1 -- 172.21.15.107:0/965663895 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f78f4014b20 con 0x7f790406b6c0 2023-12-04T05:47:55.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.793+0000 7f79027fc640 1 --2- 172.21.15.107:0/965663895 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f78f00668d0 0x7f78f0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:55.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.793+0000 7f79027fc640 1 -- 172.21.15.107:0/965663895 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f78f4084f70 con 0x7f790406b6c0 2023-12-04T05:47:55.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.793+0000 7f7909677640 1 --2- 172.21.15.107:0/965663895 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f78f00668d0 0x7f78f0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:55.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.793+0000 7f7909677640 1 --2- 172.21.15.107:0/965663895 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f78f00668d0 0x7f78f0068d90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f78f80ad000 tx=0x7f78f80a6000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:55.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.797+0000 7f79027fc640 1 -- 172.21.15.107:0/965663895 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f78f40b6280 con 0x7f790406b6c0 2023-12-04T05:47:55.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.985+0000 7f790c0dc640 1 -- 172.21.15.107:0/965663895 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7904072c00 con 0x7f790406b6c0 2023-12-04T05:47:55.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.985+0000 7f79027fc640 1 -- 172.21.15.107:0/965663895 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f78f4051fe0 con 0x7f790406b6c0 2023-12-04T05:47:55.988 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:55.988 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:55.988 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:55.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.985+0000 7f790c0dc640 1 -- 172.21.15.107:0/965663895 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f78f00668d0 msgr2=0x7f78f0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:55.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.985+0000 7f790c0dc640 1 --2- 172.21.15.107:0/965663895 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f78f00668d0 0x7f78f0068d90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f78f80ad000 tx=0x7f78f80a6000 comp rx=0 tx=0).stop 2023-12-04T05:47:55.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.985+0000 7f790c0dc640 1 -- 172.21.15.107:0/965663895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f790406b6c0 msgr2=0x7f79040c8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:55.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.985+0000 7f790c0dc640 1 --2- 172.21.15.107:0/965663895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f790406b6c0 0x7f79040c8bf0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f78f4007e40 tx=0x7f78f400e4b0 comp rx=0 tx=0).stop 2023-12-04T05:47:55.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.985+0000 7f790c0dc640 1 -- 172.21.15.107:0/965663895 shutdown_connections 2023-12-04T05:47:55.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.985+0000 7f790c0dc640 1 --2- 172.21.15.107:0/965663895 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f78f00668d0 0x7f78f0068d90 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:55.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.985+0000 7f790c0dc640 1 --2- 172.21.15.107:0/965663895 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f790415bdd0 0x7f79040c92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:55.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.985+0000 7f790c0dc640 1 --2- 172.21.15.107:0/965663895 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7904072430 0x7f7904148b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:55.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.985+0000 7f790c0dc640 1 --2- 172.21.15.107:0/965663895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f790406b6c0 0x7f79040c8bf0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:55.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.985+0000 7f790c0dc640 1 -- 172.21.15.107:0/965663895 >> 172.21.15.107:0/965663895 conn(0x7f79040663a0 msgr2=0x7f79041595f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:55.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.989+0000 7f790c0dc640 1 -- 172.21.15.107:0/965663895 shutdown_connections 2023-12-04T05:47:55.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:55.989+0000 7f790c0dc640 1 -- 172.21.15.107:0/965663895 wait complete. 2023-12-04T05:47:56.006 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:47:58.007 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:47:58.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.129+0000 7fe71783f640 1 Processor -- start 2023-12-04T05:47:58.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.129+0000 7fe71783f640 1 -- start start 2023-12-04T05:47:58.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.133+0000 7fe71783f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe7100724f0 0x7fe710158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:58.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.133+0000 7fe71783f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe7100c56f0 0x7fe710140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:58.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.133+0000 7fe71783f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe710158da0 con 0x7fe710072110 2023-12-04T05:47:58.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.133+0000 7fe71783f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7101583d0 con 0x7fe7100724f0 2023-12-04T05:47:58.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.133+0000 7fe71783f640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7100c6170 con 0x7fe7100c56f0 2023-12-04T05:47:58.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.133+0000 7fe714dda640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe7100724f0 0x7fe710158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:58.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.133+0000 7fe714dda640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe7100724f0 0x7fe710158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36194/0 (socket says 172.21.15.107:36194) 2023-12-04T05:47:58.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.133+0000 7fe714dda640 1 -- 172.21.15.107:0/2779254960 learned_addr learned my addr 172.21.15.107:0/2779254960 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:58.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.133+0000 7fe715ddc640 1 --2- 172.21.15.107:0/2779254960 >> v2:172.21.15.195:3300/0 conn(0x7fe7100c56f0 0x7fe710140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:58.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.133+0000 7fe714dda640 1 -- 172.21.15.107:0/2779254960 >> v2:172.21.15.195:3300/0 conn(0x7fe7100c56f0 msgr2=0x7fe710140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:58.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.133+0000 7fe714dda640 1 --2- 172.21.15.107:0/2779254960 >> v2:172.21.15.195:3300/0 conn(0x7fe7100c56f0 0x7fe710140830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:58.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.133+0000 7fe714dda640 1 -- 172.21.15.107:0/2779254960 >> v1:172.21.15.195:6789/0 conn(0x7fe710072110 legacy=0x7fe7100c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:47:58.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.133+0000 7fe714dda640 1 -- 172.21.15.107:0/2779254960 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe710157da0 con 0x7fe7100724f0 2023-12-04T05:47:58.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.133+0000 7fe715ddc640 1 --2- 172.21.15.107:0/2779254960 >> v2:172.21.15.195:3300/0 conn(0x7fe7100c56f0 0x7fe710140830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:47:58.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.133+0000 7fe714dda640 1 --2- 172.21.15.107:0/2779254960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7100724f0 0x7fe710158860 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe7100c9080 tx=0x7fe70001ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79efa881e28300e6 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:58.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.133+0000 7fe707fff640 1 -- 172.21.15.107:0/2779254960 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7000a89b0 con 0x7fe7100724f0 2023-12-04T05:47:58.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.133+0000 7fe707fff640 1 -- 172.21.15.107:0/2779254960 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7000a8b10 con 0x7fe7100724f0 2023-12-04T05:47:58.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.133+0000 7fe707fff640 1 -- 172.21.15.107:0/2779254960 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7000a8ea0 con 0x7fe7100724f0 2023-12-04T05:47:58.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe71783f640 1 -- 172.21.15.107:0/2779254960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7100724f0 msgr2=0x7fe710158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:58.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe71783f640 1 --2- 172.21.15.107:0/2779254960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7100724f0 0x7fe710158860 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe7100c9080 tx=0x7fe70001ecb0 comp rx=0 tx=0).stop 2023-12-04T05:47:58.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe71783f640 1 -- 172.21.15.107:0/2779254960 shutdown_connections 2023-12-04T05:47:58.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe71783f640 1 --2- 172.21.15.107:0/2779254960 >> v2:172.21.15.195:3300/0 conn(0x7fe7100c56f0 0x7fe710140830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:58.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe71783f640 1 --2- 172.21.15.107:0/2779254960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7100724f0 0x7fe710158860 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:58.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe71783f640 1 -- 172.21.15.107:0/2779254960 >> 172.21.15.107:0/2779254960 conn(0x7fe7100c7ff0 msgr2=0x7fe7100c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:58.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe71783f640 1 -- 172.21.15.107:0/2779254960 shutdown_connections 2023-12-04T05:47:58.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe71783f640 1 -- 172.21.15.107:0/2779254960 wait complete. 2023-12-04T05:47:58.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe71783f640 1 Processor -- start 2023-12-04T05:47:58.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe71783f640 1 -- start start 2023-12-04T05:47:58.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe71783f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe710072110 0x7fe7100c65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:58.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe71783f640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7100724f0 0x7fe71015a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:58.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe71783f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7100c56f0 0x7fe710146b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:58.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe71783f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7101ee0f0 con 0x7fe7100c56f0 2023-12-04T05:47:58.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe71783f640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7100ca890 con 0x7fe7100724f0 2023-12-04T05:47:58.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe71783f640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7101ed800 con 0x7fe710072110 2023-12-04T05:47:58.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe715ddc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7100c56f0 0x7fe710146b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:58.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe715ddc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7100c56f0 0x7fe710146b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36210/0 (socket says 172.21.15.107:36210) 2023-12-04T05:47:58.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe715ddc640 1 -- 172.21.15.107:0/384108283 learned_addr learned my addr 172.21.15.107:0/384108283 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:47:58.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe715ddc640 1 -- 172.21.15.107:0/384108283 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe710072110 msgr2=0x7fe7100c65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:58.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe7155db640 1 --2- 172.21.15.107:0/384108283 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe710072110 0x7fe7100c65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:58.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe714dda640 1 --2- 172.21.15.107:0/384108283 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7100724f0 0x7fe71015a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:58.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe715ddc640 1 --2- 172.21.15.107:0/384108283 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe710072110 0x7fe7100c65a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:58.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe715ddc640 1 -- 172.21.15.107:0/384108283 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7100724f0 msgr2=0x7fe71015a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:58.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe715ddc640 1 --2- 172.21.15.107:0/384108283 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7100724f0 0x7fe71015a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:58.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe715ddc640 1 -- 172.21.15.107:0/384108283 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7000094d0 con 0x7fe7100c56f0 2023-12-04T05:47:58.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe715ddc640 1 --2- 172.21.15.107:0/384108283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7100c56f0 0x7fe710146b30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fe70800cd50 tx=0x7fe70800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:58.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe705ffb640 1 -- 172.21.15.107:0/384108283 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70800f680 con 0x7fe7100c56f0 2023-12-04T05:47:58.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe705ffb640 1 -- 172.21.15.107:0/384108283 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70800f800 con 0x7fe7100c56f0 2023-12-04T05:47:58.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe71783f640 1 -- 172.21.15.107:0/384108283 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7101f1810 con 0x7fe7100c56f0 2023-12-04T05:47:58.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe705ffb640 1 -- 172.21.15.107:0/384108283 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70800fbb0 con 0x7fe7100c56f0 2023-12-04T05:47:58.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.137+0000 7fe71783f640 1 -- 172.21.15.107:0/384108283 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7101edd80 con 0x7fe7100c56f0 2023-12-04T05:47:58.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.141+0000 7fe71783f640 1 -- 172.21.15.107:0/384108283 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7101ee260 con 0x7fe7100c56f0 2023-12-04T05:47:58.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.141+0000 7fe705ffb640 1 -- 172.21.15.107:0/384108283 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe70800b040 con 0x7fe7100c56f0 2023-12-04T05:47:58.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.141+0000 7fe705ffb640 1 --2- 172.21.15.107:0/384108283 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe6f40668d0 0x7fe6f4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:47:58.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.141+0000 7fe705ffb640 1 -- 172.21.15.107:0/384108283 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fe7080850c0 con 0x7fe7100c56f0 2023-12-04T05:47:58.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.141+0000 7fe7155db640 1 --2- 172.21.15.107:0/384108283 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe6f40668d0 0x7fe6f4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:47:58.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.141+0000 7fe7155db640 1 --2- 172.21.15.107:0/384108283 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe6f40668d0 0x7fe6f4068d90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe70c003d60 tx=0x7fe70c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:47:58.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.145+0000 7fe705ffb640 1 -- 172.21.15.107:0/384108283 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe7080520a0 con 0x7fe7100c56f0 2023-12-04T05:47:58.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.317+0000 7fe71783f640 1 -- 172.21.15.107:0/384108283 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe710067870 con 0x7fe7100c56f0 2023-12-04T05:47:58.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.317+0000 7fe705ffb640 1 -- 172.21.15.107:0/384108283 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7fe7080556c0 con 0x7fe7100c56f0 2023-12-04T05:47:58.322 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:47:58.322 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:47:58.322 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:47:58.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.321+0000 7fe71783f640 1 -- 172.21.15.107:0/384108283 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe6f40668d0 msgr2=0x7fe6f4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:58.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.321+0000 7fe71783f640 1 --2- 172.21.15.107:0/384108283 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe6f40668d0 0x7fe6f4068d90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe70c003d60 tx=0x7fe70c009040 comp rx=0 tx=0).stop 2023-12-04T05:47:58.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.321+0000 7fe71783f640 1 -- 172.21.15.107:0/384108283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7100c56f0 msgr2=0x7fe710146b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:47:58.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.321+0000 7fe71783f640 1 --2- 172.21.15.107:0/384108283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7100c56f0 0x7fe710146b30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fe70800cd50 tx=0x7fe70800a650 comp rx=0 tx=0).stop 2023-12-04T05:47:58.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.321+0000 7fe71783f640 1 -- 172.21.15.107:0/384108283 shutdown_connections 2023-12-04T05:47:58.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.321+0000 7fe71783f640 1 --2- 172.21.15.107:0/384108283 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe6f40668d0 0x7fe6f4068d90 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:58.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.321+0000 7fe71783f640 1 --2- 172.21.15.107:0/384108283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7100c56f0 0x7fe710146b30 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:58.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.321+0000 7fe71783f640 1 --2- 172.21.15.107:0/384108283 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe7100724f0 0x7fe71015a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:58.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.321+0000 7fe71783f640 1 --2- 172.21.15.107:0/384108283 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe710072110 0x7fe7100c65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:47:58.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.321+0000 7fe71783f640 1 -- 172.21.15.107:0/384108283 >> 172.21.15.107:0/384108283 conn(0x7fe7100c7ff0 msgr2=0x7fe710155c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:47:58.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.321+0000 7fe71783f640 1 -- 172.21.15.107:0/384108283 shutdown_connections 2023-12-04T05:47:58.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:47:58.321+0000 7fe71783f640 1 -- 172.21.15.107:0/384108283 wait complete. 2023-12-04T05:47:58.339 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:48:00.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:48:00.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.473+0000 7f6ad74e7640 1 Processor -- start 2023-12-04T05:48:00.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.473+0000 7f6ad74e7640 1 -- start start 2023-12-04T05:48:00.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.473+0000 7f6ad74e7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6ad00724f0 0x7f6ad0158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:00.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.473+0000 7f6ad74e7640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ad00c56f0 0x7f6ad0140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:00.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.473+0000 7f6ad74e7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad0158da0 con 0x7f6ad0072110 2023-12-04T05:48:00.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.473+0000 7f6ad74e7640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad01583d0 con 0x7f6ad00724f0 2023-12-04T05:48:00.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.473+0000 7f6ad74e7640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad00c6170 con 0x7f6ad00c56f0 2023-12-04T05:48:00.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.473+0000 7f6ad5283640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6ad0072110 0x7f6ad00c7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54980/0 (socket says 172.21.15.107:54980) 2023-12-04T05:48:00.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.473+0000 7f6ad5283640 1 -- 172.21.15.107:0/2813160761 learned_addr learned my addr 172.21.15.107:0/2813160761 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:48:00.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.473+0000 7f6ad4a82640 1 --2- 172.21.15.107:0/2813160761 >> v2:172.21.15.107:3300/0 conn(0x7f6ad00724f0 0x7f6ad0158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:00.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.473+0000 7f6ad5a84640 1 --2- 172.21.15.107:0/2813160761 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ad00c56f0 0x7f6ad0140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:00.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.473+0000 7f6ac7fff640 1 -- 172.21.15.107:0/2813160761 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778380845 0 0) 0x7f6acc001120 con 0x7f6ad0072110 2023-12-04T05:48:00.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.473+0000 7f6ac7fff640 1 -- 172.21.15.107:0/2813160761 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6abc003530 con 0x7f6ad0072110 2023-12-04T05:48:00.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.473+0000 7f6ad4a82640 1 -- 172.21.15.107:0/2813160761 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ad00c56f0 msgr2=0x7f6ad0140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:00.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.473+0000 7f6ad4a82640 1 --2- 172.21.15.107:0/2813160761 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ad00c56f0 0x7f6ad0140830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:00.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.473+0000 7f6ad4a82640 1 -- 172.21.15.107:0/2813160761 >> v1:172.21.15.107:6789/0 conn(0x7f6ad0072110 legacy=0x7f6ad00c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:00.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.473+0000 7f6ad4a82640 1 -- 172.21.15.107:0/2813160761 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad0157da0 con 0x7f6ad00724f0 2023-12-04T05:48:00.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad4a82640 1 --2- 172.21.15.107:0/2813160761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ad00724f0 0x7f6ad0158860 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f6ac00097c0 tx=0x7f6ac001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5e41252c95151c server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:00.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ac7fff640 1 -- 172.21.15.107:0/2813160761 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac00ab600 con 0x7f6ad00724f0 2023-12-04T05:48:00.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ac7fff640 1 -- 172.21.15.107:0/2813160761 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac00a7050 con 0x7f6ad00724f0 2023-12-04T05:48:00.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ac7fff640 1 -- 172.21.15.107:0/2813160761 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac00ab980 con 0x7f6ad00724f0 2023-12-04T05:48:00.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad74e7640 1 -- 172.21.15.107:0/2813160761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ad00724f0 msgr2=0x7f6ad0158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:00.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad74e7640 1 --2- 172.21.15.107:0/2813160761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ad00724f0 0x7f6ad0158860 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f6ac00097c0 tx=0x7f6ac001c480 comp rx=0 tx=0).stop 2023-12-04T05:48:00.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad74e7640 1 -- 172.21.15.107:0/2813160761 shutdown_connections 2023-12-04T05:48:00.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad74e7640 1 --2- 172.21.15.107:0/2813160761 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ad00c56f0 0x7f6ad0140830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:00.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad74e7640 1 --2- 172.21.15.107:0/2813160761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ad00724f0 0x7f6ad0158860 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:00.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad74e7640 1 -- 172.21.15.107:0/2813160761 >> 172.21.15.107:0/2813160761 conn(0x7f6ad00c7ff0 msgr2=0x7f6ad00c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:00.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad74e7640 1 -- 172.21.15.107:0/2813160761 shutdown_connections 2023-12-04T05:48:00.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad74e7640 1 -- 172.21.15.107:0/2813160761 wait complete. 2023-12-04T05:48:00.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad74e7640 1 Processor -- start 2023-12-04T05:48:00.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad74e7640 1 -- start start 2023-12-04T05:48:00.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad74e7640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ad0072110 0x7f6ad0071310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:00.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad74e7640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ad00724f0 0x7f6ad0162f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:00.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad74e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ad00c56f0 0x7f6ad015ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:00.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad74e7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad013cc40 con 0x7f6ad00c56f0 2023-12-04T05:48:00.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad74e7640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad01f68e0 con 0x7f6ad0072110 2023-12-04T05:48:00.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad74e7640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad01f28f0 con 0x7f6ad00724f0 2023-12-04T05:48:00.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad5a84640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ad00c56f0 0x7f6ad015ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:00.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad5a84640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ad00c56f0 0x7f6ad015ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36216/0 (socket says 172.21.15.107:36216) 2023-12-04T05:48:00.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad5a84640 1 -- 172.21.15.107:0/1386794780 learned_addr learned my addr 172.21.15.107:0/1386794780 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:48:00.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad5283640 1 --2- 172.21.15.107:0/1386794780 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ad0072110 0x7f6ad0071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:00.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad4a82640 1 --2- 172.21.15.107:0/1386794780 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ad00724f0 0x7f6ad0162f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:00.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad5a84640 1 -- 172.21.15.107:0/1386794780 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ad00724f0 msgr2=0x7f6ad0162f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:00.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad5a84640 1 --2- 172.21.15.107:0/1386794780 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ad00724f0 0x7f6ad0162f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:00.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad5a84640 1 -- 172.21.15.107:0/1386794780 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ad0072110 msgr2=0x7f6ad0071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:00.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad5a84640 1 --2- 172.21.15.107:0/1386794780 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ad0072110 0x7f6ad0071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:00.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad5a84640 1 -- 172.21.15.107:0/1386794780 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac0009420 con 0x7f6ad00c56f0 2023-12-04T05:48:00.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad5283640 1 --2- 172.21.15.107:0/1386794780 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ad0072110 0x7f6ad0071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:48:00.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad5a84640 1 --2- 172.21.15.107:0/1386794780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ad00c56f0 0x7f6ad015ab50 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8007e60 tx=0x7f6ac800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:00.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ac5ffb640 1 -- 172.21.15.107:0/1386794780 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8014420 con 0x7f6ad00c56f0 2023-12-04T05:48:00.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ac5ffb640 1 -- 172.21.15.107:0/1386794780 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac80145a0 con 0x7f6ad00c56f0 2023-12-04T05:48:00.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad74e7640 1 -- 172.21.15.107:0/1386794780 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad01f2bd0 con 0x7f6ad00c56f0 2023-12-04T05:48:00.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ac5ffb640 1 -- 172.21.15.107:0/1386794780 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8014950 con 0x7f6ad00c56f0 2023-12-04T05:48:00.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.477+0000 7f6ad74e7640 1 -- 172.21.15.107:0/1386794780 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad01f3190 con 0x7f6ad00c56f0 2023-12-04T05:48:00.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.481+0000 7f6ad74e7640 1 -- 172.21.15.107:0/1386794780 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad01f3380 con 0x7f6ad00c56f0 2023-12-04T05:48:00.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.481+0000 7f6ac5ffb640 1 -- 172.21.15.107:0/1386794780 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f6ac8014ab0 con 0x7f6ad00c56f0 2023-12-04T05:48:00.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.481+0000 7f6ac5ffb640 1 --2- 172.21.15.107:0/1386794780 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6ab40668d0 0x7f6ab4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:00.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.481+0000 7f6ac5ffb640 1 -- 172.21.15.107:0/1386794780 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f6ac8084fa0 con 0x7f6ad00c56f0 2023-12-04T05:48:00.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.481+0000 7f6ad5283640 1 --2- 172.21.15.107:0/1386794780 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6ab40668d0 0x7f6ab4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:00.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.481+0000 7f6ad5283640 1 --2- 172.21.15.107:0/1386794780 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6ab40668d0 0x7f6ab4068d90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6acc007fd0 tx=0x7f6acc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:00.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.485+0000 7f6ac5ffb640 1 -- 172.21.15.107:0/1386794780 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6ac80b6280 con 0x7f6ad00c56f0 2023-12-04T05:48:00.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~139s 2023-12-04T05:48:00.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~253s 2023-12-04T05:48:00.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~294s 2023-12-04T05:48:00.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.669+0000 7f6ad74e7640 1 -- 172.21.15.107:0/1386794780 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ad0067870 con 0x7f6ad00c56f0 2023-12-04T05:48:00.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.669+0000 7f6ac5ffb640 1 -- 172.21.15.107:0/1386794780 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f6ac8055aa0 con 0x7f6ad00c56f0 2023-12-04T05:48:00.674 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:48:00.674 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:48:00.675 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:48:00.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.673+0000 7f6ad74e7640 1 -- 172.21.15.107:0/1386794780 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6ab40668d0 msgr2=0x7f6ab4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:00.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.673+0000 7f6ad74e7640 1 --2- 172.21.15.107:0/1386794780 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6ab40668d0 0x7f6ab4068d90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6acc007fd0 tx=0x7f6acc009040 comp rx=0 tx=0).stop 2023-12-04T05:48:00.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.673+0000 7f6ad74e7640 1 -- 172.21.15.107:0/1386794780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ad00c56f0 msgr2=0x7f6ad015ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:00.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.673+0000 7f6ad74e7640 1 --2- 172.21.15.107:0/1386794780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ad00c56f0 0x7f6ad015ab50 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8007e60 tx=0x7f6ac800e4b0 comp rx=0 tx=0).stop 2023-12-04T05:48:00.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.673+0000 7f6ad74e7640 1 -- 172.21.15.107:0/1386794780 shutdown_connections 2023-12-04T05:48:00.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.673+0000 7f6ad74e7640 1 --2- 172.21.15.107:0/1386794780 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f6ab40668d0 0x7f6ab4068d90 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:00.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.673+0000 7f6ad74e7640 1 --2- 172.21.15.107:0/1386794780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ad00c56f0 0x7f6ad015ab50 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:00.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.673+0000 7f6ad74e7640 1 --2- 172.21.15.107:0/1386794780 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f6ad00724f0 0x7f6ad0162f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:00.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.673+0000 7f6ad74e7640 1 --2- 172.21.15.107:0/1386794780 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6ad0072110 0x7f6ad0071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:00.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.673+0000 7f6ad74e7640 1 -- 172.21.15.107:0/1386794780 >> 172.21.15.107:0/1386794780 conn(0x7f6ad00c7ff0 msgr2=0x7f6ad00c71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:00.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.673+0000 7f6ad74e7640 1 -- 172.21.15.107:0/1386794780 shutdown_connections 2023-12-04T05:48:00.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:00.673+0000 7f6ad74e7640 1 -- 172.21.15.107:0/1386794780 wait complete. 2023-12-04T05:48:00.692 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:48:01.322 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:48:01.326 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:48:02.693 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:48:02.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.821+0000 7fe52ee36640 1 Processor -- start 2023-12-04T05:48:02.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.821+0000 7fe52ee36640 1 -- start start 2023-12-04T05:48:02.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.825+0000 7fe52ee36640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe528072110 0x7fe528158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:02.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.825+0000 7fe52ee36640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe5280724f0 0x7fe5280c7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:02.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.825+0000 7fe52ee36640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe528158da0 con 0x7fe5280c56f0 2023-12-04T05:48:02.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.825+0000 7fe52ee36640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5281583d0 con 0x7fe528072110 2023-12-04T05:48:02.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.825+0000 7fe52ee36640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5280c6170 con 0x7fe5280724f0 2023-12-04T05:48:02.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.825+0000 7fe52cbd2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe528072110 0x7fe528158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:02.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.825+0000 7fe52d3d3640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe5280c56f0 0x7fe528140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54992/0 (socket says 172.21.15.107:54992) 2023-12-04T05:48:02.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.825+0000 7fe52d3d3640 1 -- 172.21.15.107:0/2804941013 learned_addr learned my addr 172.21.15.107:0/2804941013 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:48:02.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.825+0000 7fe527fff640 1 --2- 172.21.15.107:0/2804941013 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe5280724f0 0x7fe5280c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:02.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.825+0000 7fe52cbd2640 1 -- 172.21.15.107:0/2804941013 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe5280724f0 msgr2=0x7fe5280c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:02.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.825+0000 7fe52cbd2640 1 --2- 172.21.15.107:0/2804941013 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe5280724f0 0x7fe5280c7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:02.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.825+0000 7fe52cbd2640 1 -- 172.21.15.107:0/2804941013 >> v1:172.21.15.107:6789/0 conn(0x7fe5280c56f0 legacy=0x7fe528140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:02.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.825+0000 7fe52cbd2640 1 -- 172.21.15.107:0/2804941013 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe528157da0 con 0x7fe528072110 2023-12-04T05:48:02.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.825+0000 7fe5277fe640 1 -- 172.21.15.107:0/2804941013 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479068953 0 0) 0x7fe5200014a0 con 0x7fe5280c56f0 2023-12-04T05:48:02.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.825+0000 7fe52cbd2640 1 --2- 172.21.15.107:0/2804941013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe528072110 0x7fe528158860 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fe5180097c0 tx=0x7fe51801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bbba6488a568f26 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:02.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.825+0000 7fe5277fe640 1 -- 172.21.15.107:0/2804941013 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5180a8970 con 0x7fe528072110 2023-12-04T05:48:02.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.825+0000 7fe5277fe640 1 -- 172.21.15.107:0/2804941013 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5180a8ad0 con 0x7fe528072110 2023-12-04T05:48:02.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.825+0000 7fe5277fe640 1 -- 172.21.15.107:0/2804941013 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5180a8e60 con 0x7fe528072110 2023-12-04T05:48:02.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.825+0000 7fe52ee36640 1 -- 172.21.15.107:0/2804941013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe528072110 msgr2=0x7fe528158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:02.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.825+0000 7fe52ee36640 1 --2- 172.21.15.107:0/2804941013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe528072110 0x7fe528158860 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fe5180097c0 tx=0x7fe51801ec70 comp rx=0 tx=0).stop 2023-12-04T05:48:02.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52ee36640 1 -- 172.21.15.107:0/2804941013 shutdown_connections 2023-12-04T05:48:02.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52ee36640 1 --2- 172.21.15.107:0/2804941013 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe5280724f0 0x7fe5280c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:02.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52ee36640 1 --2- 172.21.15.107:0/2804941013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe528072110 0x7fe528158860 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:02.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52ee36640 1 -- 172.21.15.107:0/2804941013 >> 172.21.15.107:0/2804941013 conn(0x7fe5280c7ff0 msgr2=0x7fe5280c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:02.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52ee36640 1 -- 172.21.15.107:0/2804941013 shutdown_connections 2023-12-04T05:48:02.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52ee36640 1 -- 172.21.15.107:0/2804941013 wait complete. 2023-12-04T05:48:02.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52ee36640 1 Processor -- start 2023-12-04T05:48:02.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52ee36640 1 -- start start 2023-12-04T05:48:02.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52ee36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe528072110 0x7fe52806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:02.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52ee36640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe5280724f0 0x7fe5281eb710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:02.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52ee36640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5280c56f0 0x7fe5281ebc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:02.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52ee36640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5281ef7e0 con 0x7fe528072110 2023-12-04T05:48:02.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52ee36640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe528070610 con 0x7fe5280c56f0 2023-12-04T05:48:02.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52ee36640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5281ee980 con 0x7fe5280724f0 2023-12-04T05:48:02.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52cbd2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe528072110 0x7fe52806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:02.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52cbd2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe528072110 0x7fe52806d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36244/0 (socket says 172.21.15.107:36244) 2023-12-04T05:48:02.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52cbd2640 1 -- 172.21.15.107:0/1286583060 learned_addr learned my addr 172.21.15.107:0/1286583060 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:48:02.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52cbd2640 1 -- 172.21.15.107:0/1286583060 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe5280724f0 msgr2=0x7fe5281eb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:02.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52cbd2640 1 --2- 172.21.15.107:0/1286583060 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe5280724f0 0x7fe5281eb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:02.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52cbd2640 1 -- 172.21.15.107:0/1286583060 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5280c56f0 msgr2=0x7fe5281ebc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:02.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52cbd2640 1 --2- 172.21.15.107:0/1286583060 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5280c56f0 0x7fe5281ebc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:02.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52cbd2640 1 -- 172.21.15.107:0/1286583060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe518009420 con 0x7fe528072110 2023-12-04T05:48:02.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52cbd2640 1 --2- 172.21.15.107:0/1286583060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe528072110 0x7fe52806d810 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe51801d320 tx=0x7fe51801d4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:02.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe5257fa640 1 -- 172.21.15.107:0/1286583060 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5180a8970 con 0x7fe528072110 2023-12-04T05:48:02.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe5257fa640 1 -- 172.21.15.107:0/1286583060 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe518004030 con 0x7fe528072110 2023-12-04T05:48:02.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52ee36640 1 -- 172.21.15.107:0/1286583060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5281eec60 con 0x7fe528072110 2023-12-04T05:48:02.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe5257fa640 1 -- 172.21.15.107:0/1286583060 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5180043a0 con 0x7fe528072110 2023-12-04T05:48:02.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.829+0000 7fe52ee36640 1 -- 172.21.15.107:0/1286583060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5281ef220 con 0x7fe528072110 2023-12-04T05:48:02.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.833+0000 7fe52ee36640 1 -- 172.21.15.107:0/1286583060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe528067870 con 0x7fe528072110 2023-12-04T05:48:02.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.833+0000 7fe5257fa640 1 -- 172.21.15.107:0/1286583060 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe518003620 con 0x7fe528072110 2023-12-04T05:48:02.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.833+0000 7fe5257fa640 1 --2- 172.21.15.107:0/1286583060 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe5140668d0 0x7fe514068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:02.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.833+0000 7fe527fff640 1 --2- 172.21.15.107:0/1286583060 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe5140668d0 0x7fe514068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:02.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.833+0000 7fe5257fa640 1 -- 172.21.15.107:0/1286583060 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fe5181223c0 con 0x7fe528072110 2023-12-04T05:48:02.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.833+0000 7fe527fff640 1 --2- 172.21.15.107:0/1286583060 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe5140668d0 0x7fe514068d90 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe52814e100 tx=0x7fe51c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:02.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:02.837+0000 7fe5257fa640 1 -- 172.21.15.107:0/1286583060 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe5180ef390 con 0x7fe528072110 2023-12-04T05:48:03.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:03.025+0000 7fe52ee36640 1 -- 172.21.15.107:0/1286583060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe52806e6e0 con 0x7fe528072110 2023-12-04T05:48:03.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:03.025+0000 7fe5257fa640 1 -- 172.21.15.107:0/1286583060 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7fe5180f29b0 con 0x7fe528072110 2023-12-04T05:48:03.028 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:48:03.028 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:48:03.028 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:48:03.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:03.025+0000 7fe52ee36640 1 -- 172.21.15.107:0/1286583060 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe5140668d0 msgr2=0x7fe514068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:03.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:03.025+0000 7fe52ee36640 1 --2- 172.21.15.107:0/1286583060 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe5140668d0 0x7fe514068d90 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe52814e100 tx=0x7fe51c008040 comp rx=0 tx=0).stop 2023-12-04T05:48:03.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:03.025+0000 7fe52ee36640 1 -- 172.21.15.107:0/1286583060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe528072110 msgr2=0x7fe52806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:03.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:03.025+0000 7fe52ee36640 1 --2- 172.21.15.107:0/1286583060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe528072110 0x7fe52806d810 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe51801d320 tx=0x7fe51801d4b0 comp rx=0 tx=0).stop 2023-12-04T05:48:03.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:03.025+0000 7fe52ee36640 1 -- 172.21.15.107:0/1286583060 shutdown_connections 2023-12-04T05:48:03.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:03.025+0000 7fe52ee36640 1 --2- 172.21.15.107:0/1286583060 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe5140668d0 0x7fe514068d90 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:03.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:03.025+0000 7fe52ee36640 1 --2- 172.21.15.107:0/1286583060 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe5280c56f0 0x7fe5281ebc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:03.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:03.025+0000 7fe52ee36640 1 --2- 172.21.15.107:0/1286583060 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe5280724f0 0x7fe5281eb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:03.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:03.025+0000 7fe52ee36640 1 --2- 172.21.15.107:0/1286583060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe528072110 0x7fe52806d810 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:03.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:03.025+0000 7fe52ee36640 1 -- 172.21.15.107:0/1286583060 >> 172.21.15.107:0/1286583060 conn(0x7fe5280c7ff0 msgr2=0x7fe528157310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:03.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:03.029+0000 7fe52ee36640 1 -- 172.21.15.107:0/1286583060 shutdown_connections 2023-12-04T05:48:03.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:03.029+0000 7fe52ee36640 1 -- 172.21.15.107:0/1286583060 wait complete. 2023-12-04T05:48:03.048 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 50, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-12-04T05:39:59.032019+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_4288": { "addr": "172.21.15.195:6837/3412865477", "addrs": { "addrvec": [ { "addr": "172.21.15.195:6835", "nonce": 3412865477, "type": "v2" }, { "addr": "172.21.15.195:6837", "nonce": 3412865477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_5190": { "addr": "172.21.15.107:6836/2928623624", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 2928623624, "type": "v2" }, { "addr": "172.21.15.107:6836", "nonce": 2928623624, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5190, "incarnation": 45, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:rejoin", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-12-04T05:46:11.040247+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2, 3, 4 ], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 5190 } } } ], "standbys": [ { "addr": "172.21.15.107:6837/2752497349", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6835", "nonce": 2752497349, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 2752497349, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 49, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5292, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-12-04T05:48:05.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:48:05.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.177+0000 7faccf8c3640 1 Processor -- start 2023-12-04T05:48:05.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.177+0000 7faccf8c3640 1 -- start start 2023-12-04T05:48:05.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.177+0000 7faccf8c3640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7facc8151180 0x7facc80c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:05.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.177+0000 7faccf8c3640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7facc80714f0 0x7facc80718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:05.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.177+0000 7faccf8c3640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc80700c0 con 0x7facc8070380 2023-12-04T05:48:05.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.177+0000 7faccf8c3640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc8156df0 con 0x7facc80714f0 2023-12-04T05:48:05.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.177+0000 7faccf8c3640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc81570d0 con 0x7facc8151180 2023-12-04T05:48:05.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.177+0000 7faccd65f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7facc8151180 0x7facc80c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:05.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.177+0000 7faccce5e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7facc80714f0 0x7facc80718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:05.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.177+0000 7faccde60640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7facc8070380 0x7facc8070790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:37316/0 (socket says 172.21.15.107:37316) 2023-12-04T05:48:05.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.177+0000 7faccde60640 1 -- 172.21.15.107:0/501730262 learned_addr learned my addr 172.21.15.107:0/501730262 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:48:05.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.177+0000 7faccce5e640 1 -- 172.21.15.107:0/501730262 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7facc8151180 msgr2=0x7facc80c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:05.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.177+0000 7faccce5e640 1 --2- 172.21.15.107:0/501730262 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7facc8151180 0x7facc80c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:05.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.177+0000 7faccce5e640 1 -- 172.21.15.107:0/501730262 >> v1:172.21.15.195:6789/0 conn(0x7facc8070380 legacy=0x7facc8070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:05.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.177+0000 7faccce5e640 1 -- 172.21.15.107:0/501730262 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc81573b0 con 0x7facc80714f0 2023-12-04T05:48:05.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.181+0000 7faccce5e640 1 --2- 172.21.15.107:0/501730262 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7facc80714f0 0x7facc80718f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7facc40097c0 tx=0x7facc401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f1277df4eb06368 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:05.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.181+0000 7facbffff640 1 -- 172.21.15.107:0/501730262 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc40a8970 con 0x7facc80714f0 2023-12-04T05:48:05.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.181+0000 7facbffff640 1 -- 172.21.15.107:0/501730262 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc40a8ad0 con 0x7facc80714f0 2023-12-04T05:48:05.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.181+0000 7facbffff640 1 -- 172.21.15.107:0/501730262 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc40a8e60 con 0x7facc80714f0 2023-12-04T05:48:05.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.181+0000 7faccf8c3640 1 -- 172.21.15.107:0/501730262 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7facc80714f0 msgr2=0x7facc80718f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:05.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.181+0000 7faccf8c3640 1 --2- 172.21.15.107:0/501730262 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7facc80714f0 0x7facc80718f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7facc40097c0 tx=0x7facc401ec70 comp rx=0 tx=0).stop 2023-12-04T05:48:05.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.181+0000 7faccf8c3640 1 -- 172.21.15.107:0/501730262 shutdown_connections 2023-12-04T05:48:05.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.181+0000 7faccf8c3640 1 --2- 172.21.15.107:0/501730262 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7facc80714f0 0x7facc80718f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:05.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.181+0000 7faccf8c3640 1 --2- 172.21.15.107:0/501730262 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7facc8151180 0x7facc80c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:05.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.181+0000 7faccf8c3640 1 -- 172.21.15.107:0/501730262 >> 172.21.15.107:0/501730262 conn(0x7facc80c7ff0 msgr2=0x7facc80c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:05.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.181+0000 7faccf8c3640 1 -- 172.21.15.107:0/501730262 shutdown_connections 2023-12-04T05:48:05.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.181+0000 7faccf8c3640 1 -- 172.21.15.107:0/501730262 wait complete. 2023-12-04T05:48:05.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.181+0000 7faccf8c3640 1 Processor -- start 2023-12-04T05:48:05.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.181+0000 7faccf8c3640 1 -- start start 2023-12-04T05:48:05.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccf8c3640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7facc8070380 0x7facc8143270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:05.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccf8c3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facc80714f0 0x7facc814b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:05.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccf8c3640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7facc8151180 0x7facc8158e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:05.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccf8c3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc814d2e0 con 0x7facc80714f0 2023-12-04T05:48:05.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccf8c3640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc81ed6e0 con 0x7facc8070380 2023-12-04T05:48:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccf8c3640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc814ccb0 con 0x7facc8151180 2023-12-04T05:48:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccce5e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facc80714f0 0x7facc814b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccce5e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facc80714f0 0x7facc814b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36246/0 (socket says 172.21.15.107:36246) 2023-12-04T05:48:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccce5e640 1 -- 172.21.15.107:0/3163373420 learned_addr learned my addr 172.21.15.107:0/3163373420 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:48:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccde60640 1 --2- 172.21.15.107:0/3163373420 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7facc8151180 0x7facc8158e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccd65f640 1 --2- 172.21.15.107:0/3163373420 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7facc8070380 0x7facc8143270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccce5e640 1 -- 172.21.15.107:0/3163373420 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7facc8151180 msgr2=0x7facc8158e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccce5e640 1 --2- 172.21.15.107:0/3163373420 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7facc8151180 0x7facc8158e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccce5e640 1 -- 172.21.15.107:0/3163373420 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7facc8070380 msgr2=0x7facc8143270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccce5e640 1 --2- 172.21.15.107:0/3163373420 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7facc8070380 0x7facc8143270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccce5e640 1 -- 172.21.15.107:0/3163373420 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc4009420 con 0x7facc80714f0 2023-12-04T05:48:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccd65f640 1 --2- 172.21.15.107:0/3163373420 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7facc8070380 0x7facc8143270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:48:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccce5e640 1 --2- 172.21.15.107:0/3163373420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facc80714f0 0x7facc814b430 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7facc401c8a0 tx=0x7facc401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7facbdffb640 1 -- 172.21.15.107:0/3163373420 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc40b1e00 con 0x7facc80714f0 2023-12-04T05:48:05.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7facbdffb640 1 -- 172.21.15.107:0/3163373420 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc40b9410 con 0x7facc80714f0 2023-12-04T05:48:05.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7facbdffb640 1 -- 172.21.15.107:0/3163373420 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc40b97c0 con 0x7facc80714f0 2023-12-04T05:48:05.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccf8c3640 1 -- 172.21.15.107:0/3163373420 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc814bdc0 con 0x7facc80714f0 2023-12-04T05:48:05.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccf8c3640 1 -- 172.21.15.107:0/3163373420 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc814c380 con 0x7facc80714f0 2023-12-04T05:48:05.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccf8c3640 1 -- 172.21.15.107:0/3163373420 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc815b170 con 0x7facc80714f0 2023-12-04T05:48:05.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7facbdffb640 1 -- 172.21.15.107:0/3163373420 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7facc40b7040 con 0x7facc80714f0 2023-12-04T05:48:05.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7facbdffb640 1 --2- 172.21.15.107:0/3163373420 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7facac0668d0 0x7facac068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:05.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7facbdffb640 1 -- 172.21.15.107:0/3163373420 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7facc4122200 con 0x7facc80714f0 2023-12-04T05:48:05.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccd65f640 1 --2- 172.21.15.107:0/3163373420 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7facac0668d0 0x7facac068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:05.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.185+0000 7faccd65f640 1 --2- 172.21.15.107:0/3163373420 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7facac0668d0 0x7facac068d90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7facb8009a40 tx=0x7facb8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:05.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.189+0000 7facbdffb640 1 -- 172.21.15.107:0/3163373420 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7facc40ef270 con 0x7facc80714f0 2023-12-04T05:48:05.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.377+0000 7faccf8c3640 1 -- 172.21.15.107:0/3163373420 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facc814d420 con 0x7facc80714f0 2023-12-04T05:48:05.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.377+0000 7facbdffb640 1 -- 172.21.15.107:0/3163373420 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7facc40f2890 con 0x7facc80714f0 2023-12-04T05:48:05.382 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:48:05.382 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:48:05.382 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:48:05.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.381+0000 7faccf8c3640 1 -- 172.21.15.107:0/3163373420 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7facac0668d0 msgr2=0x7facac068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:05.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.381+0000 7faccf8c3640 1 --2- 172.21.15.107:0/3163373420 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7facac0668d0 0x7facac068d90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7facb8009a40 tx=0x7facb8008040 comp rx=0 tx=0).stop 2023-12-04T05:48:05.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.381+0000 7faccf8c3640 1 -- 172.21.15.107:0/3163373420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facc80714f0 msgr2=0x7facc814b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:05.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.381+0000 7faccf8c3640 1 --2- 172.21.15.107:0/3163373420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facc80714f0 0x7facc814b430 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7facc401c8a0 tx=0x7facc401ba10 comp rx=0 tx=0).stop 2023-12-04T05:48:05.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.381+0000 7faccf8c3640 1 -- 172.21.15.107:0/3163373420 shutdown_connections 2023-12-04T05:48:05.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.381+0000 7faccf8c3640 1 --2- 172.21.15.107:0/3163373420 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7facac0668d0 0x7facac068d90 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:05.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.381+0000 7faccf8c3640 1 --2- 172.21.15.107:0/3163373420 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7facc8151180 0x7facc8158e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:05.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.381+0000 7faccf8c3640 1 --2- 172.21.15.107:0/3163373420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facc80714f0 0x7facc814b430 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:05.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.381+0000 7faccf8c3640 1 --2- 172.21.15.107:0/3163373420 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7facc8070380 0x7facc8143270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:05.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.381+0000 7faccf8c3640 1 -- 172.21.15.107:0/3163373420 >> 172.21.15.107:0/3163373420 conn(0x7facc80c7ff0 msgr2=0x7facc8070760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:05.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.381+0000 7faccf8c3640 1 -- 172.21.15.107:0/3163373420 shutdown_connections 2023-12-04T05:48:05.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:05.381+0000 7faccf8c3640 1 -- 172.21.15.107:0/3163373420 wait complete. 2023-12-04T05:48:05.401 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:48:07.249 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~146s 2023-12-04T05:48:07.249 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~260s 2023-12-04T05:48:07.249 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~301s 2023-12-04T05:48:07.249 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-12-04T05:48:07.249 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:48:07.250 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-04T05:48:07.257 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T05:48:07.309 INFO:tasks.workunit.client.0.smithi107.stderr:ffsb_stats.c:248:1: fatal error: opening dependency file .deps/ffsb_stats.Tpo: Software caused connection abort 2023-12-04T05:48:07.309 INFO:tasks.workunit.client.0.smithi107.stderr:compilation terminated. 2023-12-04T05:48:07.309 INFO:tasks.workunit.client.0.smithi107.stderr:make[1]: *** [Makefile:324: ffsb_stats.o] Error 1 2023-12-04T05:48:07.310 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2023-12-04T05:48:07.310 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-12-04T05:48:07.311 INFO:tasks.workunit.client.0.smithi107.stderr:make: *** [Makefile:217: all] Error 2 2023-12-04T05:48:07.311 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2023-12-04T05:48:07.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-04T05:48:07.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:48:07.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.421+0000 7f9ced351640 1 Processor -- start 2023-12-04T05:48:07.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.421+0000 7f9ced351640 1 -- start start 2023-12-04T05:48:07.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ced351640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f9ce8071030 0x7f9ce80c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:07.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ced351640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9ce80c60e0 0x7f9ce806e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:07.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ced351640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce8152360 con 0x7f9ce8151f80 2023-12-04T05:48:07.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ced351640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ce806acf0 con 0x7f9ce80c60e0 2023-12-04T05:48:07.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ced351640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ce8071460 con 0x7f9ce8071030 2023-12-04T05:48:07.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ce77fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9ce80c60e0 0x7f9ce806e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:07.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ce77fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9ce80c60e0 0x7f9ce806e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36262/0 (socket says 172.21.15.107:36262) 2023-12-04T05:48:07.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ce77fe640 1 -- 172.21.15.107:0/1024447193 learned_addr learned my addr 172.21.15.107:0/1024447193 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:48:07.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ce67fc640 1 --2- 172.21.15.107:0/1024447193 >> v2:172.21.15.195:3300/0 conn(0x7f9ce8071030 0x7f9ce80c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:07.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ce77fe640 1 -- 172.21.15.107:0/1024447193 >> v2:172.21.15.195:3300/0 conn(0x7f9ce8071030 msgr2=0x7f9ce80c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:07.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ce77fe640 1 --2- 172.21.15.107:0/1024447193 >> v2:172.21.15.195:3300/0 conn(0x7f9ce8071030 0x7f9ce80c5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ce77fe640 1 -- 172.21.15.107:0/1024447193 >> v1:172.21.15.195:6789/0 conn(0x7f9ce8151f80 legacy=0x7f9ce806c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:07.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ce77fe640 1 -- 172.21.15.107:0/1024447193 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce8157c90 con 0x7f9ce80c60e0 2023-12-04T05:48:07.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ce77fe640 1 --2- 172.21.15.107:0/1024447193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ce80c60e0 0x7f9ce806e1f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9ce80c8ef0 tx=0x7f9ce001f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fe05176c0893cd0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:07.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ce5ffb640 1 -- 172.21.15.107:0/1024447193 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce00af400 con 0x7f9ce80c60e0 2023-12-04T05:48:07.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ce5ffb640 1 -- 172.21.15.107:0/1024447193 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce00ab050 con 0x7f9ce80c60e0 2023-12-04T05:48:07.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ce5ffb640 1 -- 172.21.15.107:0/1024447193 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce00af780 con 0x7f9ce80c60e0 2023-12-04T05:48:07.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ced351640 1 -- 172.21.15.107:0/1024447193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ce80c60e0 msgr2=0x7f9ce806e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:07.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ced351640 1 --2- 172.21.15.107:0/1024447193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ce80c60e0 0x7f9ce806e1f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9ce80c8ef0 tx=0x7f9ce001f180 comp rx=0 tx=0).stop 2023-12-04T05:48:07.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ced351640 1 -- 172.21.15.107:0/1024447193 shutdown_connections 2023-12-04T05:48:07.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ced351640 1 --2- 172.21.15.107:0/1024447193 >> v2:172.21.15.195:3300/0 conn(0x7f9ce8071030 0x7f9ce80c5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ced351640 1 --2- 172.21.15.107:0/1024447193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ce80c60e0 0x7f9ce806e1f0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ced351640 1 -- 172.21.15.107:0/1024447193 >> 172.21.15.107:0/1024447193 conn(0x7f9ce80cc490 msgr2=0x7f9ce80cc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:07.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ced351640 1 -- 172.21.15.107:0/1024447193 shutdown_connections 2023-12-04T05:48:07.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.425+0000 7f9ced351640 1 -- 172.21.15.107:0/1024447193 wait complete. 2023-12-04T05:48:07.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ced351640 1 Processor -- start 2023-12-04T05:48:07.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ced351640 1 -- start start 2023-12-04T05:48:07.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ced351640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ce8071030 0x7f9ce814c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:07.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ced351640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9ce80c60e0 0x7f9ce80c51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:07.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ced351640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9ce8151f80 0x7f9ce8149c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:07.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ced351640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce80cc150 con 0x7f9ce8071030 2023-12-04T05:48:07.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ced351640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce8148130 con 0x7f9ce8151f80 2023-12-04T05:48:07.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ced351640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce8148410 con 0x7f9ce80c60e0 2023-12-04T05:48:07.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ce6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ce8071030 0x7f9ce814c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:07.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ce6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ce8071030 0x7f9ce814c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36264/0 (socket says 172.21.15.107:36264) 2023-12-04T05:48:07.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ce6ffd640 1 -- 172.21.15.107:0/2205259227 learned_addr learned my addr 172.21.15.107:0/2205259227 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:48:07.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ce77fe640 1 --2- 172.21.15.107:0/2205259227 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9ce8151f80 0x7f9ce8149c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:07.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ce67fc640 1 --2- 172.21.15.107:0/2205259227 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9ce80c60e0 0x7f9ce80c51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:07.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ce6ffd640 1 -- 172.21.15.107:0/2205259227 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9ce80c60e0 msgr2=0x7f9ce80c51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:07.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ce6ffd640 1 --2- 172.21.15.107:0/2205259227 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9ce80c60e0 0x7f9ce80c51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ce6ffd640 1 -- 172.21.15.107:0/2205259227 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9ce8151f80 msgr2=0x7f9ce8149c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:07.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ce6ffd640 1 --2- 172.21.15.107:0/2205259227 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9ce8151f80 0x7f9ce8149c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ce6ffd640 1 -- 172.21.15.107:0/2205259227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce0009b30 con 0x7f9ce8071030 2023-12-04T05:48:07.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ce6ffd640 1 --2- 172.21.15.107:0/2205259227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ce8071030 0x7f9ce814c830 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9cd800b410 tx=0x7f9cd800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:07.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ccf7fe640 1 -- 172.21.15.107:0/2205259227 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8004020 con 0x7f9ce8071030 2023-12-04T05:48:07.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ccf7fe640 1 -- 172.21.15.107:0/2205259227 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd8004180 con 0x7f9ce8071030 2023-12-04T05:48:07.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ced351640 1 -- 172.21.15.107:0/2205259227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce80cb820 con 0x7f9ce8071030 2023-12-04T05:48:07.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ccf7fe640 1 -- 172.21.15.107:0/2205259227 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8011630 con 0x7f9ce8071030 2023-12-04T05:48:07.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ced351640 1 -- 172.21.15.107:0/2205259227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce80cbde0 con 0x7f9ce8071030 2023-12-04T05:48:07.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.429+0000 7f9ced351640 1 -- 172.21.15.107:0/2205259227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce8067ec0 con 0x7f9ce8071030 2023-12-04T05:48:07.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.433+0000 7f9ccf7fe640 1 -- 172.21.15.107:0/2205259227 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f9cd8011790 con 0x7f9ce8071030 2023-12-04T05:48:07.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.433+0000 7f9ccf7fe640 1 --2- 172.21.15.107:0/2205259227 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9cd40668d0 0x7f9cd4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:07.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.433+0000 7f9ccf7fe640 1 -- 172.21.15.107:0/2205259227 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f9cd8084b90 con 0x7f9ce8071030 2023-12-04T05:48:07.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.433+0000 7f9ce67fc640 1 --2- 172.21.15.107:0/2205259227 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9cd40668d0 0x7f9cd4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:07.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.433+0000 7f9ce67fc640 1 --2- 172.21.15.107:0/2205259227 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9cd40668d0 0x7f9cd4068d90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9ce806ebf0 tx=0x7f9cdc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:07.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.437+0000 7f9ccf7fe640 1 -- 172.21.15.107:0/2205259227 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9cd8051c50 con 0x7f9ce8071030 2023-12-04T05:48:07.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.521+0000 7f5867434640 1 Processor -- start 2023-12-04T05:48:07.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.521+0000 7f5867434640 1 -- start start 2023-12-04T05:48:07.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f5867434640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5860159f40 0x7f5860153da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:07.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f5867434640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f586006abf0 0x7f58601542e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:07.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f5867434640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58600c7430 con 0x7f586006a810 2023-12-04T05:48:07.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f5867434640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5860072680 con 0x7f5860159f40 2023-12-04T05:48:07.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f5867434640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58600c8d00 con 0x7f586006abf0 2023-12-04T05:48:07.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f58649cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5860159f40 0x7f5860153da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:07.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f58649cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5860159f40 0x7f5860153da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36266/0 (socket says 172.21.15.107:36266) 2023-12-04T05:48:07.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f58649cf640 1 -- 172.21.15.107:0/858001924 learned_addr learned my addr 172.21.15.107:0/858001924 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:48:07.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f58659d1640 1 --2- 172.21.15.107:0/858001924 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f586006abf0 0x7f58601542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:07.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f58649cf640 1 -- 172.21.15.107:0/858001924 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f586006abf0 msgr2=0x7f58601542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:07.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f58649cf640 1 --2- 172.21.15.107:0/858001924 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f586006abf0 0x7f58601542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f58649cf640 1 -- 172.21.15.107:0/858001924 >> v1:172.21.15.195:6789/0 conn(0x7f586006a810 legacy=0x7f58600c6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:07.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f58649cf640 1 -- 172.21.15.107:0/858001924 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f586015a3d0 con 0x7f5860159f40 2023-12-04T05:48:07.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f58649cf640 1 --2- 172.21.15.107:0/858001924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5860159f40 0x7f5860153da0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5850009870 tx=0x7f585001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36ddce22dc9a3ab2 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:07.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f5857fff640 1 -- 172.21.15.107:0/858001924 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58500a9400 con 0x7f5860159f40 2023-12-04T05:48:07.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f5857fff640 1 -- 172.21.15.107:0/858001924 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58500a9560 con 0x7f5860159f40 2023-12-04T05:48:07.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f5857fff640 1 -- 172.21.15.107:0/858001924 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58500a98f0 con 0x7f5860159f40 2023-12-04T05:48:07.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f5867434640 1 -- 172.21.15.107:0/858001924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5860159f40 msgr2=0x7f5860153da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:07.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f5867434640 1 --2- 172.21.15.107:0/858001924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5860159f40 0x7f5860153da0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5850009870 tx=0x7f585001c540 comp rx=0 tx=0).stop 2023-12-04T05:48:07.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f5867434640 1 -- 172.21.15.107:0/858001924 shutdown_connections 2023-12-04T05:48:07.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f5867434640 1 --2- 172.21.15.107:0/858001924 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f586006abf0 0x7f58601542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f5867434640 1 --2- 172.21.15.107:0/858001924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5860159f40 0x7f5860153da0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f5867434640 1 -- 172.21.15.107:0/858001924 >> 172.21.15.107:0/858001924 conn(0x7f58600cc440 msgr2=0x7f58600cc850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:07.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f5867434640 1 -- 172.21.15.107:0/858001924 shutdown_connections 2023-12-04T05:48:07.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.525+0000 7f5867434640 1 -- 172.21.15.107:0/858001924 wait complete. 2023-12-04T05:48:07.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f5867434640 1 Processor -- start 2023-12-04T05:48:07.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f5867434640 1 -- start start 2023-12-04T05:48:07.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f5867434640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f586006a810 0x7f586014b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:07.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f5867434640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f586006abf0 0x7f58601f4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:07.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f5867434640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58601f5210 0x7f58601f5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:07.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f5867434640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5860143d70 con 0x7f58601f5210 2023-12-04T05:48:07.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f5867434640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58601f4840 con 0x7f586006abf0 2023-12-04T05:48:07.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f5867434640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58601f56c0 con 0x7f586006a810 2023-12-04T05:48:07.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f58659d1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58601f5210 0x7f58601f5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:07.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f58659d1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58601f5210 0x7f58601f5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57514/0 (socket says 172.21.15.107:57514) 2023-12-04T05:48:07.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f58659d1640 1 -- 172.21.15.107:0/3590432017 learned_addr learned my addr 172.21.15.107:0/3590432017 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:48:07.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f58651d0640 1 --2- 172.21.15.107:0/3590432017 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f586006a810 0x7f586014b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:07.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f58649cf640 1 --2- 172.21.15.107:0/3590432017 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f586006abf0 0x7f58601f4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:07.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f58659d1640 1 -- 172.21.15.107:0/3590432017 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f586006a810 msgr2=0x7f586014b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:07.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f58659d1640 1 --2- 172.21.15.107:0/3590432017 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f586006a810 0x7f586014b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f58659d1640 1 -- 172.21.15.107:0/3590432017 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f586006abf0 msgr2=0x7f58601f4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:07.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f58659d1640 1 --2- 172.21.15.107:0/3590432017 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f586006abf0 0x7f58601f4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f58659d1640 1 -- 172.21.15.107:0/3590432017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58500094d0 con 0x7f58601f5210 2023-12-04T05:48:07.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f58659d1640 1 --2- 172.21.15.107:0/3590432017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58601f5210 0x7f58601f5c60 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5858007e40 tx=0x7f585800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:07.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f5855ffb640 1 -- 172.21.15.107:0/3590432017 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5858014620 con 0x7f58601f5210 2023-12-04T05:48:07.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f5855ffb640 1 -- 172.21.15.107:0/3590432017 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f585800ee90 con 0x7f58601f5210 2023-12-04T05:48:07.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f5867434640 1 -- 172.21.15.107:0/3590432017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5860153320 con 0x7f58601f5210 2023-12-04T05:48:07.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f5855ffb640 1 -- 172.21.15.107:0/3590432017 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58580149e0 con 0x7f58601f5210 2023-12-04T05:48:07.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.529+0000 7f5867434640 1 -- 172.21.15.107:0/3590432017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58601f3c30 con 0x7f58601f5210 2023-12-04T05:48:07.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.533+0000 7f5867434640 1 -- 172.21.15.107:0/3590432017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58601f49b0 con 0x7f58601f5210 2023-12-04T05:48:07.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.533+0000 7f5855ffb640 1 -- 172.21.15.107:0/3590432017 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f5858014b40 con 0x7f58601f5210 2023-12-04T05:48:07.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.533+0000 7f5855ffb640 1 --2- 172.21.15.107:0/3590432017 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5844066900 0x7f5844068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:07.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.533+0000 7f5855ffb640 1 -- 172.21.15.107:0/3590432017 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f5858084f40 con 0x7f58601f5210 2023-12-04T05:48:07.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.533+0000 7f58651d0640 1 --2- 172.21.15.107:0/3590432017 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5844066900 0x7f5844068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:07.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.533+0000 7f58651d0640 1 --2- 172.21.15.107:0/3590432017 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5844066900 0x7f5844068dc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f585c000f70 tx=0x7f585c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:07.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.537+0000 7f5855ffb640 1 -- 172.21.15.107:0/3590432017 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5858051fb0 con 0x7f58601f5210 2023-12-04T05:48:07.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.593+0000 7f9ced351640 1 -- 172.21.15.107:0/2205259227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ce8071940 con 0x7f9ce8071030 2023-12-04T05:48:07.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.597+0000 7f9ccf7fe640 1 -- 172.21.15.107:0/2205259227 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+17315 (secure 0 0 0) 0x7f9cd8015020 con 0x7f9ce8071030 2023-12-04T05:48:07.600 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:48:07.600 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"fsid":"603e856b-31ac-475a-8f35-21ab7faadd48","created":"2023-12-04T05:39:34.921690+0000","modified":"2023-12-04T05:45:54.798928+0000","last_up_change":"2023-12-04T05:39:46.235790+0000","last_in_change":"2023-12-04T05:39:37.587639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T05:39:47.567702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T05:39:57.644286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T05:39:58.029897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ab7111c-0ae8-4d41-b6db-e3743614322c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6817","nonce":768714279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6821","nonce":768714279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6829","nonce":768714279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6825","nonce":768714279}]},"public_addr":"172.21.15.107:6817/768714279","cluster_addr":"172.21.15.107:6821/768714279","heartbeat_back_addr":"172.21.15.107:6829/768714279","heartbeat_front_addr":"172.21.15.107:6825/768714279","state":["exists","up"]},{"osd":1,"uuid":"baef7a1f-8026-4fab-9e70-48465271034e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6801","nonce":2496355981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6803","nonce":2496355981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6807","nonce":2496355981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6805","nonce":2496355981}]},"public_addr":"172.21.15.107:6801/2496355981","cluster_addr":"172.21.15.107:6803/2496355981","heartbeat_back_addr":"172.21.15.107:6807/2496355981","heartbeat_front_addr":"172.21.15.107:6805/2496355981","state":["exists","up"]},{"osd":2,"uuid":"90e8cf24-7b54-484a-9d85-b44a9d3dfeab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6809","nonce":3097298224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6811","nonce":3097298224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6815","nonce":3097298224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6813","nonce":3097298224}]},"public_addr":"172.21.15.107:6809/3097298224","cluster_addr":"172.21.15.107:6811/3097298224","heartbeat_back_addr":"172.21.15.107:6815/3097298224","heartbeat_front_addr":"172.21.15.107:6813/3097298224","state":["exists","up"]},{"osd":3,"uuid":"54658731-2b9a-4d5e-ab13-7c1be7d34e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6820","nonce":3773076671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6824","nonce":3773076671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6831","nonce":3773076671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6828","nonce":3773076671}]},"public_addr":"172.21.15.107:6820/3773076671","cluster_addr":"172.21.15.107:6824/3773076671","heartbeat_back_addr":"172.21.15.107:6831/3773076671","heartbeat_front_addr":"172.21.15.107:6828/3773076671","state":["exists","up"]},{"osd":4,"uuid":"9a9dea13-8afe-485d-aa43-45eda691179b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6801","nonce":2747823542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6803","nonce":2747823542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6807","nonce":2747823542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6805","nonce":2747823542}]},"public_addr":"172.21.15.195:6801/2747823542","cluster_addr":"172.21.15.195:6803/2747823542","heartbeat_back_addr":"172.21.15.195:6807/2747823542","heartbeat_front_addr":"172.21.15.195:6805/2747823542","state":["exists","up"]},{"osd":5,"uuid":"db840d83-5a7d-468a-8aa2-1997945847c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6825","nonce":1484264540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6827","nonce":1484264540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6831","nonce":1484264540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6829","nonce":1484264540}]},"public_addr":"172.21.15.195:6825/1484264540","cluster_addr":"172.21.15.195:6827/1484264540","heartbeat_back_addr":"172.21.15.195:6831/1484264540","heartbeat_front_addr":"172.21.15.195:6829/1484264540","state":["exists","up"]},{"osd":6,"uuid":"38b5f165-bf91-467e-a615-4b4784337689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6811","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6813","nonce":2638471412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6817","nonce":2638471412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6823","nonce":2638471412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6821","nonce":2638471412}]},"public_addr":"172.21.15.195:6813/2638471412","cluster_addr":"172.21.15.195:6817/2638471412","heartbeat_back_addr":"172.21.15.195:6823/2638471412","heartbeat_front_addr":"172.21.15.195:6821/2638471412","state":["exists","up"]},{"osd":7,"uuid":"e5b3c596-494b-4df3-9437-bc2d52168ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6809","nonce":768189614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6812","nonce":768189614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6820","nonce":768189614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6816","nonce":768189614}]},"public_addr":"172.21.15.195:6809/768189614","cluster_addr":"172.21.15.195:6812/768189614","heartbeat_back_addr":"172.21.15.195:6820/768189614","heartbeat_front_addr":"172.21.15.195:6816/768189614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.209077+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.086601+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.221568+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.238524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.059477+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.148422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.079867+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.064916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6839/3319682203":"2023-12-05T05:45:54.798833+0000","172.21.15.107:6838/3319682203":"2023-12-05T05:45:54.798833+0000","172.21.15.195:6836/2222928136":"2023-12-05T05:45:09.028534+0000","172.21.15.195:6833/569093002":"2023-12-05T05:43:14.736286+0000","172.21.15.195:6832/569093002":"2023-12-05T05:43:14.736286+0000","172.21.15.195:6834/2222928136":"2023-12-05T05:45:09.028534+0000","172.21.15.195:6836/3769196693":"2023-12-05T05:42:14.714882+0000","172.21.15.195:6834/3769196693":"2023-12-05T05:42:14.714882+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T05:48:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.601+0000 7f9ced351640 1 -- 172.21.15.107:0/2205259227 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9cd40668d0 msgr2=0x7f9cd4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.601+0000 7f9ced351640 1 --2- 172.21.15.107:0/2205259227 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9cd40668d0 0x7f9cd4068d90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9ce806ebf0 tx=0x7f9cdc008040 comp rx=0 tx=0).stop 2023-12-04T05:48:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.601+0000 7f9ced351640 1 -- 172.21.15.107:0/2205259227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ce8071030 msgr2=0x7f9ce814c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.601+0000 7f9ced351640 1 --2- 172.21.15.107:0/2205259227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ce8071030 0x7f9ce814c830 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9cd800b410 tx=0x7f9cd800b8d0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.601+0000 7f9ced351640 1 -- 172.21.15.107:0/2205259227 shutdown_connections 2023-12-04T05:48:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.601+0000 7f9ced351640 1 --2- 172.21.15.107:0/2205259227 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f9cd40668d0 0x7f9cd4068d90 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.601+0000 7f9ced351640 1 --2- 172.21.15.107:0/2205259227 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9ce8151f80 0x7f9ce8149c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.601+0000 7f9ced351640 1 --2- 172.21.15.107:0/2205259227 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f9ce80c60e0 0x7f9ce80c51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.601+0000 7f9ced351640 1 --2- 172.21.15.107:0/2205259227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ce8071030 0x7f9ce814c830 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.601+0000 7f9ced351640 1 -- 172.21.15.107:0/2205259227 >> 172.21.15.107:0/2205259227 conn(0x7f9ce80cc490 msgr2=0x7f9ce806be80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.601+0000 7f9ced351640 1 -- 172.21.15.107:0/2205259227 shutdown_connections 2023-12-04T05:48:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.601+0000 7f9ced351640 1 -- 172.21.15.107:0/2205259227 wait complete. 2023-12-04T05:48:07.622 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-04T05:39:47.567702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-04T05:39:57.644286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-04T05:39:58.029897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-04T05:48:07.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-04T05:48:07.656 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi107 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47c1c547372ed5bff7984647dc200bac0e019caf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2023-12-04T05:48:07.738 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=00ae224516a3476ea76a04ad4ac52aeb Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi107 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47c1c547372ed5bff7984647dc200bac0e019caf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2023-12-04T05:48:07.741 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2023-12-04T05:48:07.751 INFO:tasks.mds_thrash:joining mds_thrasher 2023-12-04T05:48:07.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.725+0000 7f5867434640 1 -- 172.21.15.107:0/3590432017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5860067ec0 con 0x7f58601f5210 2023-12-04T05:48:07.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.725+0000 7f5855ffb640 1 -- 172.21.15.107:0/3590432017 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f58580555d0 con 0x7f58601f5210 2023-12-04T05:48:07.760 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:48:07.760 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:48:07.761 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:48:07.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.725+0000 7f5867434640 1 -- 172.21.15.107:0/3590432017 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5844066900 msgr2=0x7f5844068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:07.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.725+0000 7f5867434640 1 --2- 172.21.15.107:0/3590432017 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5844066900 0x7f5844068dc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f585c000f70 tx=0x7f585c009040 comp rx=0 tx=0).stop 2023-12-04T05:48:07.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.725+0000 7f5867434640 1 -- 172.21.15.107:0/3590432017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58601f5210 msgr2=0x7f58601f5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:07.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.725+0000 7f5867434640 1 --2- 172.21.15.107:0/3590432017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58601f5210 0x7f58601f5c60 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5858007e40 tx=0x7f585800e4b0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.725+0000 7f5867434640 1 -- 172.21.15.107:0/3590432017 shutdown_connections 2023-12-04T05:48:07.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.725+0000 7f5867434640 1 --2- 172.21.15.107:0/3590432017 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5844066900 0x7f5844068dc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.725+0000 7f5867434640 1 --2- 172.21.15.107:0/3590432017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58601f5210 0x7f58601f5c60 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.725+0000 7f5867434640 1 --2- 172.21.15.107:0/3590432017 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f586006abf0 0x7f58601f4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.725+0000 7f5867434640 1 --2- 172.21.15.107:0/3590432017 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f586006a810 0x7f586014b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.725+0000 7f5867434640 1 -- 172.21.15.107:0/3590432017 >> 172.21.15.107:0/3590432017 conn(0x7f58600cc440 msgr2=0x7f58600c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:07.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.725+0000 7f5867434640 1 -- 172.21.15.107:0/3590432017 shutdown_connections 2023-12-04T05:48:07.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.729+0000 7f5867434640 1 -- 172.21.15.107:0/3590432017 wait complete. 2023-12-04T05:48:07.764 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:48:07.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.813+0000 7f61a2e1e640 1 Processor -- start 2023-12-04T05:48:07.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.813+0000 7f61a2e1e640 1 -- start start 2023-12-04T05:48:07.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f61a2e1e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f619c072110 0x7f619c158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:07.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f61a2e1e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f619c0724f0 0x7f619c0c7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:07.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f61a2e1e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f619c158da0 con 0x7f619c0c56f0 2023-12-04T05:48:07.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f61a2e1e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f619c1583d0 con 0x7f619c072110 2023-12-04T05:48:07.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f61a2e1e640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f619c0c6170 con 0x7f619c0724f0 2023-12-04T05:48:07.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f61a0bba640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f619c072110 0x7f619c158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:07.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f61a13bb640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f619c0c56f0 0x7f619c140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60422/0 (socket says 172.21.15.107:60422) 2023-12-04T05:48:07.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f61a13bb640 1 -- 172.21.15.107:0/471766150 learned_addr learned my addr 172.21.15.107:0/471766150 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:48:07.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f619bfff640 1 --2- 172.21.15.107:0/471766150 >> v2:172.21.15.195:3300/0 conn(0x7f619c0724f0 0x7f619c0c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:07.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f61a0bba640 1 -- 172.21.15.107:0/471766150 >> v2:172.21.15.195:3300/0 conn(0x7f619c0724f0 msgr2=0x7f619c0c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:07.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f61a0bba640 1 --2- 172.21.15.107:0/471766150 >> v2:172.21.15.195:3300/0 conn(0x7f619c0724f0 0x7f619c0c7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f61a0bba640 1 -- 172.21.15.107:0/471766150 >> v1:172.21.15.107:6789/0 conn(0x7f619c0c56f0 legacy=0x7f619c140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:07.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f61a0bba640 1 -- 172.21.15.107:0/471766150 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619c157da0 con 0x7f619c072110 2023-12-04T05:48:07.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f619b7fe640 1 -- 172.21.15.107:0/471766150 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051162031 0 0) 0x7f61940014a0 con 0x7f619c0c56f0 2023-12-04T05:48:07.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f619bfff640 1 --2- 172.21.15.107:0/471766150 >> v2:172.21.15.195:3300/0 conn(0x7f619c0724f0 0x7f619c0c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:48:07.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f61a0bba640 1 --2- 172.21.15.107:0/471766150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c072110 0x7f619c158860 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f619c06b870 tx=0x7f618c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ac5a37cdfcca36f server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:07.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f619b7fe640 1 -- 172.21.15.107:0/471766150 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618c0a8950 con 0x7f619c072110 2023-12-04T05:48:07.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f619b7fe640 1 -- 172.21.15.107:0/471766150 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f618c0a8ab0 con 0x7f619c072110 2023-12-04T05:48:07.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f619b7fe640 1 -- 172.21.15.107:0/471766150 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618c0a8e40 con 0x7f619c072110 2023-12-04T05:48:07.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f61a2e1e640 1 -- 172.21.15.107:0/471766150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c072110 msgr2=0x7f619c158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:07.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f61a2e1e640 1 --2- 172.21.15.107:0/471766150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c072110 0x7f619c158860 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f619c06b870 tx=0x7f618c01ec50 comp rx=0 tx=0).stop 2023-12-04T05:48:07.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f61a2e1e640 1 -- 172.21.15.107:0/471766150 shutdown_connections 2023-12-04T05:48:07.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f61a2e1e640 1 --2- 172.21.15.107:0/471766150 >> v2:172.21.15.195:3300/0 conn(0x7f619c0724f0 0x7f619c0c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f61a2e1e640 1 --2- 172.21.15.107:0/471766150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c072110 0x7f619c158860 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f61a2e1e640 1 -- 172.21.15.107:0/471766150 >> 172.21.15.107:0/471766150 conn(0x7f619c0c7ff0 msgr2=0x7f619c0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:07.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f61a2e1e640 1 -- 172.21.15.107:0/471766150 shutdown_connections 2023-12-04T05:48:07.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.817+0000 7f61a2e1e640 1 -- 172.21.15.107:0/471766150 wait complete. 2023-12-04T05:48:07.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f61a2e1e640 1 Processor -- start 2023-12-04T05:48:07.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f61a2e1e640 1 -- start start 2023-12-04T05:48:07.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f61a2e1e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f619c072110 0x7f619c1ed470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:07.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f61a2e1e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c0724f0 0x7f619c15b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:07.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f61a2e1e640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f619c0c56f0 0x7f619c153810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:07.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f61a2e1e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c1553d0 con 0x7f619c0724f0 2023-12-04T05:48:07.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f61a2e1e640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c154850 con 0x7f619c0c56f0 2023-12-04T05:48:07.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f61a2e1e640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619c154b30 con 0x7f619c072110 2023-12-04T05:48:07.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f619bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c0724f0 0x7f619c15b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:07.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f619bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c0724f0 0x7f619c15b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57542/0 (socket says 172.21.15.107:57542) 2023-12-04T05:48:07.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f619bfff640 1 -- 172.21.15.107:0/2706365689 learned_addr learned my addr 172.21.15.107:0/2706365689 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:48:07.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f619bfff640 1 -- 172.21.15.107:0/2706365689 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f619c072110 msgr2=0x7f619c1ed470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:07.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f619bfff640 1 --2- 172.21.15.107:0/2706365689 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f619c072110 0x7f619c1ed470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f619bfff640 1 -- 172.21.15.107:0/2706365689 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f619c0c56f0 msgr2=0x7f619c153810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:07.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f619bfff640 1 --2- 172.21.15.107:0/2706365689 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f619c0c56f0 0x7f619c153810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f619bfff640 1 -- 172.21.15.107:0/2706365689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f618c009420 con 0x7f619c0724f0 2023-12-04T05:48:07.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f619bfff640 1 --2- 172.21.15.107:0/2706365689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c0724f0 0x7f619c15b8e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f619000c7f0 tx=0x7f619000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:07.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f61997fa640 1 -- 172.21.15.107:0/2706365689 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619000cea0 con 0x7f619c0724f0 2023-12-04T05:48:07.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f61997fa640 1 -- 172.21.15.107:0/2706365689 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6190007a30 con 0x7f619c0724f0 2023-12-04T05:48:07.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f61a2e1e640 1 -- 172.21.15.107:0/2706365689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f619c154e10 con 0x7f619c0724f0 2023-12-04T05:48:07.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f61997fa640 1 -- 172.21.15.107:0/2706365689 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6190007de0 con 0x7f619c0724f0 2023-12-04T05:48:07.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f61a2e1e640 1 -- 172.21.15.107:0/2706365689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f619c1532f0 con 0x7f619c0724f0 2023-12-04T05:48:07.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f61a2e1e640 1 -- 172.21.15.107:0/2706365689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f619c153150 con 0x7f619c0724f0 2023-12-04T05:48:07.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.821+0000 7f61997fa640 1 -- 172.21.15.107:0/2706365689 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f61900113f0 con 0x7f619c0724f0 2023-12-04T05:48:07.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.825+0000 7f61997fa640 1 --2- 172.21.15.107:0/2706365689 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f61880668d0 0x7f6188068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:07.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.825+0000 7f61997fa640 1 -- 172.21.15.107:0/2706365689 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f61900847c0 con 0x7f619c0724f0 2023-12-04T05:48:07.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.825+0000 7f61a0bba640 1 --2- 172.21.15.107:0/2706365689 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f61880668d0 0x7f6188068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:07.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.825+0000 7f61a0bba640 1 --2- 172.21.15.107:0/2706365689 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f61880668d0 0x7f6188068d90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f618c0b2000 tx=0x7f618c004740 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:07.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.829+0000 7f61997fa640 1 -- 172.21.15.107:0/2706365689 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f619c153150 con 0x7f619c0724f0 2023-12-04T05:48:07.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.981+0000 7f61a2e1e640 1 -- 172.21.15.107:0/2706365689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f619c153150 con 0x7f619c0724f0 2023-12-04T05:48:07.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.981+0000 7f61997fa640 1 -- 172.21.15.107:0/2706365689 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f619c153150 con 0x7f619c0724f0 2023-12-04T05:48:07.986 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-12-04T05:48:07.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.985+0000 7f61a2e1e640 1 -- 172.21.15.107:0/2706365689 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f61880668d0 msgr2=0x7f6188068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:07.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.985+0000 7f61a2e1e640 1 --2- 172.21.15.107:0/2706365689 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f61880668d0 0x7f6188068d90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f618c0b2000 tx=0x7f618c004740 comp rx=0 tx=0).stop 2023-12-04T05:48:07.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.985+0000 7f61a2e1e640 1 -- 172.21.15.107:0/2706365689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c0724f0 msgr2=0x7f619c15b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:07.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.985+0000 7f61a2e1e640 1 --2- 172.21.15.107:0/2706365689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c0724f0 0x7f619c15b8e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f619000c7f0 tx=0x7f619000ccb0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.985+0000 7f61a2e1e640 1 -- 172.21.15.107:0/2706365689 shutdown_connections 2023-12-04T05:48:07.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.985+0000 7f61a2e1e640 1 --2- 172.21.15.107:0/2706365689 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f61880668d0 0x7f6188068d90 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.985+0000 7f61a2e1e640 1 --2- 172.21.15.107:0/2706365689 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f619c0c56f0 0x7f619c153810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.985+0000 7f61a2e1e640 1 --2- 172.21.15.107:0/2706365689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c0724f0 0x7f619c15b8e0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.985+0000 7f61a2e1e640 1 --2- 172.21.15.107:0/2706365689 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f619c072110 0x7f619c1ed470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:07.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.985+0000 7f61a2e1e640 1 -- 172.21.15.107:0/2706365689 >> 172.21.15.107:0/2706365689 conn(0x7f619c0c7ff0 msgr2=0x7f619c0c3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:07.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.985+0000 7f61a2e1e640 1 -- 172.21.15.107:0/2706365689 shutdown_connections 2023-12-04T05:48:07.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:07.985+0000 7f61a2e1e640 1 -- 172.21.15.107:0/2706365689 wait complete. 2023-12-04T05:48:08.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-04T05:48:08.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.165+0000 7f1eab19c640 1 Processor -- start 2023-12-04T05:48:08.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.165+0000 7f1eab19c640 1 -- start start 2023-12-04T05:48:08.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.165+0000 7f1eab19c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1ea415d700 0x7f1ea415dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:08.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.165+0000 7f1eab19c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea415e020 con 0x7f1ea406b3a0 2023-12-04T05:48:08.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.165+0000 7f1eab19c640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea4071130 con 0x7f1ea40c6380 2023-12-04T05:48:08.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.165+0000 7f1eab19c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ea4071410 con 0x7f1ea415d700 2023-12-04T05:48:08.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.165+0000 7f1ea8f38640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1ea415d700 0x7f1ea415dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:08.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.165+0000 7f1ea9739640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1ea406b3a0 0x7f1ea40c5260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60438/0 (socket says 172.21.15.107:60438) 2023-12-04T05:48:08.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.165+0000 7f1ea8f38640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1ea415d700 0x7f1ea415dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57552/0 (socket says 172.21.15.107:57552) 2023-12-04T05:48:08.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.165+0000 7f1ea9739640 1 -- 172.21.15.107:0/899253337 learned_addr learned my addr 172.21.15.107:0/899253337 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:48:08.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.165+0000 7f1ea8f38640 1 -- 172.21.15.107:0/899253337 >> v1:172.21.15.195:6789/0 conn(0x7f1ea40c6380 legacy=0x7f1ea406ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:08.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.165+0000 7f1ea8f38640 1 -- 172.21.15.107:0/899253337 >> v1:172.21.15.107:6789/0 conn(0x7f1ea406b3a0 legacy=0x7f1ea40c5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:08.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.165+0000 7f1ea8f38640 1 -- 172.21.15.107:0/899253337 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea40716f0 con 0x7f1ea415d700 2023-12-04T05:48:08.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.165+0000 7f1ea37fe640 1 -- 172.21.15.107:0/899253337 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024139793 0 0) 0x7f1ea415e020 con 0x7f1ea406b3a0 2023-12-04T05:48:08.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.169+0000 7f1ea8f38640 1 --2- 172.21.15.107:0/899253337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea415d700 0x7f1ea415dae0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1e940097c0 tx=0x7f1e9401ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60315c6b9091647d server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:08.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.169+0000 7f1ea37fe640 1 -- 172.21.15.107:0/899253337 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e940a8930 con 0x7f1ea415d700 2023-12-04T05:48:08.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.169+0000 7f1ea37fe640 1 -- 172.21.15.107:0/899253337 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e940a8a90 con 0x7f1ea415d700 2023-12-04T05:48:08.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.169+0000 7f1ea37fe640 1 -- 172.21.15.107:0/899253337 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e940a8e20 con 0x7f1ea415d700 2023-12-04T05:48:08.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.169+0000 7f1eab19c640 1 -- 172.21.15.107:0/899253337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea415d700 msgr2=0x7f1ea415dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:08.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.169+0000 7f1eab19c640 1 --2- 172.21.15.107:0/899253337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea415d700 0x7f1ea415dae0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1e940097c0 tx=0x7f1e9401ec30 comp rx=0 tx=0).stop 2023-12-04T05:48:08.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.169+0000 7f1eab19c640 1 -- 172.21.15.107:0/899253337 shutdown_connections 2023-12-04T05:48:08.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.169+0000 7f1eab19c640 1 --2- 172.21.15.107:0/899253337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea415d700 0x7f1ea415dae0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.169+0000 7f1eab19c640 1 -- 172.21.15.107:0/899253337 >> 172.21.15.107:0/899253337 conn(0x7f1ea406e6f0 msgr2=0x7f1ea40c8580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:08.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.169+0000 7f1eab19c640 1 -- 172.21.15.107:0/899253337 shutdown_connections 2023-12-04T05:48:08.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.169+0000 7f1eab19c640 1 -- 172.21.15.107:0/899253337 wait complete. 2023-12-04T05:48:08.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.169+0000 7f1eab19c640 1 Processor -- start 2023-12-04T05:48:08.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.169+0000 7f1eab19c640 1 -- start start 2023-12-04T05:48:08.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1eab19c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea406b3a0 0x7f1ea415b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:08.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1eab19c640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ea40c6380 0x7f1ea41566f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:08.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1eab19c640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1ea415d700 0x7f1ea4156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:08.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1eab19c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea40c2850 con 0x7f1ea406b3a0 2023-12-04T05:48:08.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1eab19c640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea41f3d20 con 0x7f1ea40c6380 2023-12-04T05:48:08.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1eab19c640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea40c1db0 con 0x7f1ea415d700 2023-12-04T05:48:08.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1ea8f38640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea406b3a0 0x7f1ea415b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:08.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1ea8f38640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea406b3a0 0x7f1ea415b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57554/0 (socket says 172.21.15.107:57554) 2023-12-04T05:48:08.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1ea8f38640 1 -- 172.21.15.107:0/3056574099 learned_addr learned my addr 172.21.15.107:0/3056574099 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:48:08.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1ea8f38640 1 -- 172.21.15.107:0/3056574099 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1ea415d700 msgr2=0x7f1ea4156c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:08.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1ea8f38640 1 --2- 172.21.15.107:0/3056574099 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1ea415d700 0x7f1ea4156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1ea8f38640 1 -- 172.21.15.107:0/3056574099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ea40c6380 msgr2=0x7f1ea41566f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:08.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1ea3fff640 1 --2- 172.21.15.107:0/3056574099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ea40c6380 0x7f1ea41566f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:08.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1ea8f38640 1 --2- 172.21.15.107:0/3056574099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ea40c6380 0x7f1ea41566f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1ea8f38640 1 -- 172.21.15.107:0/3056574099 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e94009420 con 0x7f1ea406b3a0 2023-12-04T05:48:08.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1ea8f38640 1 --2- 172.21.15.107:0/3056574099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea406b3a0 0x7f1ea415b350 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1e9401b840 tx=0x7f1e9401b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:08.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1ea17fa640 1 -- 172.21.15.107:0/3056574099 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e940b0e40 con 0x7f1ea406b3a0 2023-12-04T05:48:08.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1ea17fa640 1 -- 172.21.15.107:0/3056574099 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e940b9410 con 0x7f1ea406b3a0 2023-12-04T05:48:08.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1eab19c640 1 -- 172.21.15.107:0/3056574099 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea41f07d0 con 0x7f1ea406b3a0 2023-12-04T05:48:08.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1ea17fa640 1 -- 172.21.15.107:0/3056574099 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e940b97c0 con 0x7f1ea406b3a0 2023-12-04T05:48:08.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1eab19c640 1 -- 172.21.15.107:0/3056574099 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea41efa10 con 0x7f1ea406b3a0 2023-12-04T05:48:08.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1eab19c640 1 -- 172.21.15.107:0/3056574099 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ea4067ec0 con 0x7f1ea406b3a0 2023-12-04T05:48:08.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1ea17fa640 1 -- 172.21.15.107:0/3056574099 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f1e940b7040 con 0x7f1ea406b3a0 2023-12-04T05:48:08.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1ea17fa640 1 --2- 172.21.15.107:0/3056574099 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1e900668d0 0x7f1e90068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:08.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1ea17fa640 1 -- 172.21.15.107:0/3056574099 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f1e94122370 con 0x7f1ea406b3a0 2023-12-04T05:48:08.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1ea3fff640 1 --2- 172.21.15.107:0/3056574099 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1e900668d0 0x7f1e90068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:08.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.173+0000 7f1ea3fff640 1 --2- 172.21.15.107:0/3056574099 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1e900668d0 0x7f1e90068d90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1ea415d470 tx=0x7f1e98009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:08.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.177+0000 7f1ea17fa640 1 -- 172.21.15.107:0/3056574099 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1e940ef350 con 0x7f1ea406b3a0 2023-12-04T05:48:08.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.329+0000 7f1eab19c640 1 -- 172.21.15.107:0/3056574099 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1ea4067ec0 con 0x7f1ea406b3a0 2023-12-04T05:48:08.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.329+0000 7f1ea17fa640 1 -- 172.21.15.107:0/3056574099 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1e940b50a0 con 0x7f1ea406b3a0 2023-12-04T05:48:08.331 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-12-04T05:48:08.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.329+0000 7f1eab19c640 1 -- 172.21.15.107:0/3056574099 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1e900668d0 msgr2=0x7f1e90068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:08.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.329+0000 7f1eab19c640 1 --2- 172.21.15.107:0/3056574099 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1e900668d0 0x7f1e90068d90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1ea415d470 tx=0x7f1e98009040 comp rx=0 tx=0).stop 2023-12-04T05:48:08.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.329+0000 7f1eab19c640 1 -- 172.21.15.107:0/3056574099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea406b3a0 msgr2=0x7f1ea415b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:08.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.329+0000 7f1eab19c640 1 --2- 172.21.15.107:0/3056574099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea406b3a0 0x7f1ea415b350 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1e9401b840 tx=0x7f1e9401b9d0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.329+0000 7f1eab19c640 1 -- 172.21.15.107:0/3056574099 shutdown_connections 2023-12-04T05:48:08.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.329+0000 7f1eab19c640 1 --2- 172.21.15.107:0/3056574099 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f1e900668d0 0x7f1e90068d90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.329+0000 7f1eab19c640 1 --2- 172.21.15.107:0/3056574099 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1ea415d700 0x7f1ea4156c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.329+0000 7f1eab19c640 1 --2- 172.21.15.107:0/3056574099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1ea40c6380 0x7f1ea41566f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.329+0000 7f1eab19c640 1 --2- 172.21.15.107:0/3056574099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea406b3a0 0x7f1ea415b350 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.329+0000 7f1eab19c640 1 -- 172.21.15.107:0/3056574099 >> 172.21.15.107:0/3056574099 conn(0x7f1ea406e6f0 msgr2=0x7f1ea40c3720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:08.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.329+0000 7f1eab19c640 1 -- 172.21.15.107:0/3056574099 shutdown_connections 2023-12-04T05:48:08.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.329+0000 7f1eab19c640 1 -- 172.21.15.107:0/3056574099 wait complete. 2023-12-04T05:48:08.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-04T05:48:08.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.457+0000 7fe8a32bc640 1 Processor -- start 2023-12-04T05:48:08.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.457+0000 7fe8a32bc640 1 -- start start 2023-12-04T05:48:08.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a32bc640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe89c072110 0x7fe89c158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:08.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a32bc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe89c0724f0 0x7fe89c0c7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:08.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a32bc640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe89c0c56f0 0x7fe89c0c7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:08.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a32bc640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89c0c7ea0 con 0x7fe89c0724f0 2023-12-04T05:48:08.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a32bc640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89c158da0 con 0x7fe89c0c56f0 2023-12-04T05:48:08.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a32bc640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c1583d0 con 0x7fe89c072110 2023-12-04T05:48:08.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a0857640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe89c0724f0 0x7fe89c0c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:08.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a1058640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe89c072110 0x7fe89c158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:08.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a0857640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe89c0724f0 0x7fe89c0c7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57568/0 (socket says 172.21.15.107:57568) 2023-12-04T05:48:08.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a1058640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe89c072110 0x7fe89c158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3301/0 says I am v2:172.21.15.107:53468/0 (socket says 172.21.15.107:53468) 2023-12-04T05:48:08.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a0857640 1 -- 172.21.15.107:0/689834529 learned_addr learned my addr 172.21.15.107:0/689834529 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:48:08.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a1859640 1 --2- 172.21.15.107:0/689834529 >> v2:172.21.15.195:3300/0 conn(0x7fe89c0c56f0 0x7fe89c0c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:08.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a0857640 1 -- 172.21.15.107:0/689834529 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe89c072110 msgr2=0x7fe89c158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:08.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a0857640 1 --2- 172.21.15.107:0/689834529 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe89c072110 0x7fe89c158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a0857640 1 -- 172.21.15.107:0/689834529 >> v2:172.21.15.195:3300/0 conn(0x7fe89c0c56f0 msgr2=0x7fe89c0c7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:08.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a0857640 1 --2- 172.21.15.107:0/689834529 >> v2:172.21.15.195:3300/0 conn(0x7fe89c0c56f0 0x7fe89c0c7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a0857640 1 -- 172.21.15.107:0/689834529 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c0c6170 con 0x7fe89c0724f0 2023-12-04T05:48:08.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a0857640 1 --2- 172.21.15.107:0/689834529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe89c0724f0 0x7fe89c0c7420 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fe89c15c430 tx=0x7fe88c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac928860de4a2a3 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:08.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe893fff640 1 -- 172.21.15.107:0/689834529 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88c0ac410 con 0x7fe89c0724f0 2023-12-04T05:48:08.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe893fff640 1 -- 172.21.15.107:0/689834529 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe88c0ac570 con 0x7fe89c0724f0 2023-12-04T05:48:08.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe893fff640 1 -- 172.21.15.107:0/689834529 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88c0ac900 con 0x7fe89c0724f0 2023-12-04T05:48:08.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a32bc640 1 -- 172.21.15.107:0/689834529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe89c0724f0 msgr2=0x7fe89c0c7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:08.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a32bc640 1 --2- 172.21.15.107:0/689834529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe89c0724f0 0x7fe89c0c7420 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fe89c15c430 tx=0x7fe88c01c500 comp rx=0 tx=0).stop 2023-12-04T05:48:08.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a32bc640 1 -- 172.21.15.107:0/689834529 shutdown_connections 2023-12-04T05:48:08.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a32bc640 1 --2- 172.21.15.107:0/689834529 >> v2:172.21.15.195:3300/0 conn(0x7fe89c0c56f0 0x7fe89c0c7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a32bc640 1 --2- 172.21.15.107:0/689834529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe89c0724f0 0x7fe89c0c7420 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a32bc640 1 --2- 172.21.15.107:0/689834529 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe89c072110 0x7fe89c158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a32bc640 1 -- 172.21.15.107:0/689834529 >> 172.21.15.107:0/689834529 conn(0x7fe89c0c7ff0 msgr2=0x7fe89c0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:08.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a32bc640 1 -- 172.21.15.107:0/689834529 shutdown_connections 2023-12-04T05:48:08.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.461+0000 7fe8a32bc640 1 -- 172.21.15.107:0/689834529 wait complete. 2023-12-04T05:48:08.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a32bc640 1 Processor -- start 2023-12-04T05:48:08.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a32bc640 1 -- start start 2023-12-04T05:48:08.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a32bc640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe89c072110 0x7fe89c06dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:08.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a32bc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe89c0724f0 0x7fe89c137eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:08.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a32bc640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe89c0c56f0 0x7fe89c1383f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:08.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a32bc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c1f5f70 con 0x7fe89c0724f0 2023-12-04T05:48:08.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a32bc640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c1f3420 con 0x7fe89c0c56f0 2023-12-04T05:48:08.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a32bc640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c1f36e0 con 0x7fe89c072110 2023-12-04T05:48:08.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a0857640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe89c0724f0 0x7fe89c137eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:08.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a0857640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe89c0724f0 0x7fe89c137eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57576/0 (socket says 172.21.15.107:57576) 2023-12-04T05:48:08.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a0857640 1 -- 172.21.15.107:0/2222846052 learned_addr learned my addr 172.21.15.107:0/2222846052 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:48:08.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a1058640 1 --2- 172.21.15.107:0/2222846052 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe89c072110 0x7fe89c06dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:08.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a0857640 1 -- 172.21.15.107:0/2222846052 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe89c072110 msgr2=0x7fe89c06dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:08.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a0857640 1 --2- 172.21.15.107:0/2222846052 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe89c072110 0x7fe89c06dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a1859640 1 --2- 172.21.15.107:0/2222846052 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe89c0c56f0 0x7fe89c1383f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:08.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a0857640 1 -- 172.21.15.107:0/2222846052 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe89c0c56f0 msgr2=0x7fe89c1383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:08.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a0857640 1 --2- 172.21.15.107:0/2222846052 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe89c0c56f0 0x7fe89c1383f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a0857640 1 -- 172.21.15.107:0/2222846052 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe88c0094d0 con 0x7fe89c0724f0 2023-12-04T05:48:08.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a1058640 1 --2- 172.21.15.107:0/2222846052 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe89c072110 0x7fe89c06dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:48:08.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a0857640 1 --2- 172.21.15.107:0/2222846052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe89c0724f0 0x7fe89c137eb0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fe88c01abb0 tx=0x7fe88c0192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:08.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe891ffb640 1 -- 172.21.15.107:0/2222846052 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88c0b65f0 con 0x7fe89c0724f0 2023-12-04T05:48:08.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe891ffb640 1 -- 172.21.15.107:0/2222846052 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe88c0b6750 con 0x7fe89c0724f0 2023-12-04T05:48:08.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a32bc640 1 -- 172.21.15.107:0/2222846052 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe89c1f6180 con 0x7fe89c0724f0 2023-12-04T05:48:08.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe891ffb640 1 -- 172.21.15.107:0/2222846052 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88c0b6b00 con 0x7fe89c0724f0 2023-12-04T05:48:08.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a32bc640 1 -- 172.21.15.107:0/2222846052 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89c1f6740 con 0x7fe89c0724f0 2023-12-04T05:48:08.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe8a32bc640 1 -- 172.21.15.107:0/2222846052 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89c067870 con 0x7fe89c0724f0 2023-12-04T05:48:08.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe891ffb640 1 -- 172.21.15.107:0/2222846052 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe88c0b6c60 con 0x7fe89c0724f0 2023-12-04T05:48:08.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.465+0000 7fe891ffb640 1 --2- 172.21.15.107:0/2222846052 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe880066900 0x7fe880068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:08.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.469+0000 7fe891ffb640 1 -- 172.21.15.107:0/2222846052 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fe88c11f190 con 0x7fe89c0724f0 2023-12-04T05:48:08.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.469+0000 7fe8a1058640 1 --2- 172.21.15.107:0/2222846052 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe880066900 0x7fe880068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:08.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.469+0000 7fe8a1058640 1 --2- 172.21.15.107:0/2222846052 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe880066900 0x7fe880068dc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe898007520 tx=0x7fe89800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:08.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.473+0000 7fe891ffb640 1 -- 172.21.15.107:0/2222846052 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe88c0ec320 con 0x7fe89c0724f0 2023-12-04T05:48:08.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.625+0000 7fe8a32bc640 1 -- 172.21.15.107:0/2222846052 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe89c1572c0 con 0x7fe89c0724f0 2023-12-04T05:48:08.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.625+0000 7fe891ffb640 1 -- 172.21.15.107:0/2222846052 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe88c0ef940 con 0x7fe89c0724f0 2023-12-04T05:48:08.629 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-12-04T05:48:08.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.629+0000 7fe8a32bc640 1 -- 172.21.15.107:0/2222846052 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe880066900 msgr2=0x7fe880068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:08.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.629+0000 7fe8a32bc640 1 --2- 172.21.15.107:0/2222846052 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe880066900 0x7fe880068dc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe898007520 tx=0x7fe89800b040 comp rx=0 tx=0).stop 2023-12-04T05:48:08.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.629+0000 7fe8a32bc640 1 -- 172.21.15.107:0/2222846052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe89c0724f0 msgr2=0x7fe89c137eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:08.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.629+0000 7fe8a32bc640 1 --2- 172.21.15.107:0/2222846052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe89c0724f0 0x7fe89c137eb0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fe88c01abb0 tx=0x7fe88c0192a0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.629+0000 7fe8a32bc640 1 -- 172.21.15.107:0/2222846052 shutdown_connections 2023-12-04T05:48:08.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.629+0000 7fe8a32bc640 1 --2- 172.21.15.107:0/2222846052 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe880066900 0x7fe880068dc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.629+0000 7fe8a32bc640 1 --2- 172.21.15.107:0/2222846052 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe89c0c56f0 0x7fe89c1383f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.629+0000 7fe8a32bc640 1 --2- 172.21.15.107:0/2222846052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe89c0724f0 0x7fe89c137eb0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.629+0000 7fe8a32bc640 1 --2- 172.21.15.107:0/2222846052 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe89c072110 0x7fe89c06dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.629+0000 7fe8a32bc640 1 -- 172.21.15.107:0/2222846052 >> 172.21.15.107:0/2222846052 conn(0x7fe89c0c7ff0 msgr2=0x7fe89c0c95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:08.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.629+0000 7fe8a32bc640 1 -- 172.21.15.107:0/2222846052 shutdown_connections 2023-12-04T05:48:08.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.629+0000 7fe8a32bc640 1 -- 172.21.15.107:0/2222846052 wait complete. 2023-12-04T05:48:08.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-04T05:48:08.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.761+0000 7fe9fd85c640 1 Processor -- start 2023-12-04T05:48:08.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.761+0000 7fe9fd85c640 1 -- start start 2023-12-04T05:48:08.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.761+0000 7fe9fd85c640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe9f80724f0 0x7fe9f8158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:08.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.761+0000 7fe9fd85c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe9f80c56f0 0x7fe9f8140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:08.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.761+0000 7fe9fd85c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f8158da0 con 0x7fe9f8072110 2023-12-04T05:48:08.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.761+0000 7fe9fd85c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9f81583d0 con 0x7fe9f80c56f0 2023-12-04T05:48:08.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.761+0000 7fe9fd85c640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9f80c6170 con 0x7fe9f80724f0 2023-12-04T05:48:08.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.761+0000 7fe9f6ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe9f8072110 0x7fe9f80c7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60448/0 (socket says 172.21.15.107:60448) 2023-12-04T05:48:08.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.761+0000 7fe9f6ffd640 1 -- 172.21.15.107:0/2519753583 learned_addr learned my addr 172.21.15.107:0/2519753583 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:48:08.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.761+0000 7fe9f77fe640 1 --2- 172.21.15.107:0/2519753583 >> v2:172.21.15.107:3300/0 conn(0x7fe9f80c56f0 0x7fe9f8140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:08.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.761+0000 7fe9f67fc640 1 --2- 172.21.15.107:0/2519753583 >> v2:172.21.15.195:3300/0 conn(0x7fe9f80724f0 0x7fe9f8158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:08.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.761+0000 7fe9f5ffb640 1 -- 172.21.15.107:0/2519753583 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792649197 0 0) 0x7fe9e80014a0 con 0x7fe9f8072110 2023-12-04T05:48:08.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.761+0000 7fe9f5ffb640 1 -- 172.21.15.107:0/2519753583 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9e0003530 con 0x7fe9f8072110 2023-12-04T05:48:08.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.761+0000 7fe9f77fe640 1 -- 172.21.15.107:0/2519753583 >> v2:172.21.15.195:3300/0 conn(0x7fe9f80724f0 msgr2=0x7fe9f8158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:08.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.761+0000 7fe9f77fe640 1 --2- 172.21.15.107:0/2519753583 >> v2:172.21.15.195:3300/0 conn(0x7fe9f80724f0 0x7fe9f8158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.761+0000 7fe9f77fe640 1 -- 172.21.15.107:0/2519753583 >> v1:172.21.15.107:6789/0 conn(0x7fe9f8072110 legacy=0x7fe9f80c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:08.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.761+0000 7fe9f77fe640 1 -- 172.21.15.107:0/2519753583 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f8157da0 con 0x7fe9f80c56f0 2023-12-04T05:48:08.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.765+0000 7fe9f77fe640 1 --2- 172.21.15.107:0/2519753583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9f80c56f0 0x7fe9f8140830 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fe9f806b870 tx=0x7fe9f001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4066678684950909 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:08.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.765+0000 7fe9f5ffb640 1 -- 172.21.15.107:0/2519753583 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f00a8950 con 0x7fe9f80c56f0 2023-12-04T05:48:08.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.765+0000 7fe9f5ffb640 1 -- 172.21.15.107:0/2519753583 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f00a8ab0 con 0x7fe9f80c56f0 2023-12-04T05:48:08.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.765+0000 7fe9f5ffb640 1 -- 172.21.15.107:0/2519753583 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f00a8e40 con 0x7fe9f80c56f0 2023-12-04T05:48:08.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.765+0000 7fe9fd85c640 1 -- 172.21.15.107:0/2519753583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9f80c56f0 msgr2=0x7fe9f8140830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:08.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.765+0000 7fe9fd85c640 1 --2- 172.21.15.107:0/2519753583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9f80c56f0 0x7fe9f8140830 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fe9f806b870 tx=0x7fe9f001ec50 comp rx=0 tx=0).stop 2023-12-04T05:48:08.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.765+0000 7fe9fd85c640 1 -- 172.21.15.107:0/2519753583 shutdown_connections 2023-12-04T05:48:08.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.765+0000 7fe9fd85c640 1 --2- 172.21.15.107:0/2519753583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9f80c56f0 0x7fe9f8140830 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.765+0000 7fe9fd85c640 1 --2- 172.21.15.107:0/2519753583 >> v2:172.21.15.195:3300/0 conn(0x7fe9f80724f0 0x7fe9f8158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.765+0000 7fe9fd85c640 1 -- 172.21.15.107:0/2519753583 >> 172.21.15.107:0/2519753583 conn(0x7fe9f80c7ff0 msgr2=0x7fe9f80c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:08.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.765+0000 7fe9fd85c640 1 -- 172.21.15.107:0/2519753583 shutdown_connections 2023-12-04T05:48:08.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.765+0000 7fe9fd85c640 1 -- 172.21.15.107:0/2519753583 wait complete. 2023-12-04T05:48:08.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.765+0000 7fe9fd85c640 1 Processor -- start 2023-12-04T05:48:08.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.765+0000 7fe9fd85c640 1 -- start start 2023-12-04T05:48:08.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9fd85c640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe9f8072110 0x7fe9f806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:08.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9fd85c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9f80724f0 0x7fe9f8154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:08.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9fd85c640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe9f80c56f0 0x7fe9f81ef960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:08.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9fd85c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f8070070 con 0x7fe9f80724f0 2023-12-04T05:48:08.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9fd85c640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f8070350 con 0x7fe9f80c56f0 2023-12-04T05:48:08.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9fd85c640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f80711a0 con 0x7fe9f8072110 2023-12-04T05:48:08.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9f67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9f80724f0 0x7fe9f8154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:08.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9f67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9f80724f0 0x7fe9f8154f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57592/0 (socket says 172.21.15.107:57592) 2023-12-04T05:48:08.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9f67fc640 1 -- 172.21.15.107:0/765212610 learned_addr learned my addr 172.21.15.107:0/765212610 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:48:08.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9f67fc640 1 -- 172.21.15.107:0/765212610 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe9f8072110 msgr2=0x7fe9f806d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:08.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9f67fc640 1 --2- 172.21.15.107:0/765212610 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe9f8072110 0x7fe9f806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9f67fc640 1 -- 172.21.15.107:0/765212610 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe9f80c56f0 msgr2=0x7fe9f81ef960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:08.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9f77fe640 1 --2- 172.21.15.107:0/765212610 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe9f80c56f0 0x7fe9f81ef960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:08.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9f67fc640 1 --2- 172.21.15.107:0/765212610 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe9f80c56f0 0x7fe9f81ef960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9f67fc640 1 -- 172.21.15.107:0/765212610 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f0009420 con 0x7fe9f80724f0 2023-12-04T05:48:08.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9f67fc640 1 --2- 172.21.15.107:0/765212610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9f80724f0 0x7fe9f8154f70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec00d7d0 tx=0x7fe9ec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:08.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9dffff640 1 -- 172.21.15.107:0/765212610 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec018070 con 0x7fe9f80724f0 2023-12-04T05:48:08.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9dffff640 1 -- 172.21.15.107:0/765212610 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9ec004020 con 0x7fe9f80724f0 2023-12-04T05:48:08.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9fd85c640 1 -- 172.21.15.107:0/765212610 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9f8071480 con 0x7fe9f80724f0 2023-12-04T05:48:08.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9dffff640 1 -- 172.21.15.107:0/765212610 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec0041d0 con 0x7fe9f80724f0 2023-12-04T05:48:08.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9fd85c640 1 -- 172.21.15.107:0/765212610 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f8071a40 con 0x7fe9f80724f0 2023-12-04T05:48:08.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9fd85c640 1 -- 172.21.15.107:0/765212610 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9f8067870 con 0x7fe9f80724f0 2023-12-04T05:48:08.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9dffff640 1 -- 172.21.15.107:0/765212610 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe9ec004330 con 0x7fe9f80724f0 2023-12-04T05:48:08.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9dffff640 1 --2- 172.21.15.107:0/765212610 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe9e40668d0 0x7fe9e4068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:08.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9dffff640 1 -- 172.21.15.107:0/765212610 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fe9ec084530 con 0x7fe9f80724f0 2023-12-04T05:48:08.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9f6ffd640 1 --2- 172.21.15.107:0/765212610 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe9e40668d0 0x7fe9e4068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:08.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.769+0000 7fe9f6ffd640 1 --2- 172.21.15.107:0/765212610 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe9e40668d0 0x7fe9e4068d90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8005c10 tx=0x7fe9e80014e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:08.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.773+0000 7fe9dffff640 1 -- 172.21.15.107:0/765212610 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe9ec051510 con 0x7fe9f80724f0 2023-12-04T05:48:08.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.929+0000 7fe9fd85c640 1 -- 172.21.15.107:0/765212610 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe9f8156cc0 con 0x7fe9f80724f0 2023-12-04T05:48:08.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.929+0000 7fe9dffff640 1 -- 172.21.15.107:0/765212610 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v27) v1 ==== 80+0+494 (secure 0 0 0) 0x7fe9ec054b30 con 0x7fe9f80724f0 2023-12-04T05:48:08.932 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6839/3319682203 2023-12-05T05:45:54.798833+0000 2023-12-04T05:48:08.932 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6838/3319682203 2023-12-05T05:45:54.798833+0000 2023-12-04T05:48:08.932 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.195:6836/2222928136 2023-12-05T05:45:09.028534+0000 2023-12-04T05:48:08.933 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.195:6833/569093002 2023-12-05T05:43:14.736286+0000 2023-12-04T05:48:08.933 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.195:6832/569093002 2023-12-05T05:43:14.736286+0000 2023-12-04T05:48:08.933 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.195:6834/2222928136 2023-12-05T05:45:09.028534+0000 2023-12-04T05:48:08.933 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.195:6836/3769196693 2023-12-05T05:42:14.714882+0000 2023-12-04T05:48:08.933 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.195:6834/3769196693 2023-12-05T05:42:14.714882+0000 2023-12-04T05:48:08.933 INFO:teuthology.orchestra.run.smithi107.stderr:listed 8 entries 2023-12-04T05:48:08.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.933+0000 7fe9fd85c640 1 -- 172.21.15.107:0/765212610 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe9e40668d0 msgr2=0x7fe9e4068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:08.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.933+0000 7fe9fd85c640 1 --2- 172.21.15.107:0/765212610 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe9e40668d0 0x7fe9e4068d90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8005c10 tx=0x7fe9e80014e0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.933+0000 7fe9fd85c640 1 -- 172.21.15.107:0/765212610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9f80724f0 msgr2=0x7fe9f8154f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:08.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.933+0000 7fe9fd85c640 1 --2- 172.21.15.107:0/765212610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9f80724f0 0x7fe9f8154f70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec00d7d0 tx=0x7fe9ec00dc90 comp rx=0 tx=0).stop 2023-12-04T05:48:08.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.933+0000 7fe9fd85c640 1 -- 172.21.15.107:0/765212610 shutdown_connections 2023-12-04T05:48:08.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.933+0000 7fe9fd85c640 1 --2- 172.21.15.107:0/765212610 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe9e40668d0 0x7fe9e4068d90 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.933+0000 7fe9fd85c640 1 --2- 172.21.15.107:0/765212610 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe9f80c56f0 0x7fe9f81ef960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.933+0000 7fe9fd85c640 1 --2- 172.21.15.107:0/765212610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9f80724f0 0x7fe9f8154f70 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.933+0000 7fe9fd85c640 1 --2- 172.21.15.107:0/765212610 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe9f8072110 0x7fe9f806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:08.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.933+0000 7fe9fd85c640 1 -- 172.21.15.107:0/765212610 >> 172.21.15.107:0/765212610 conn(0x7fe9f80c7ff0 msgr2=0x7fe9f80c6900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:08.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.933+0000 7fe9fd85c640 1 -- 172.21.15.107:0/765212610 shutdown_connections 2023-12-04T05:48:08.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:08.933+0000 7fe9fd85c640 1 -- 172.21.15.107:0/765212610 wait complete. 2023-12-04T05:48:08.950 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:48:08.950 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-04T05:48:08.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T05:48:09.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.117+0000 7f572e71e640 1 Processor -- start 2023-12-04T05:48:09.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.117+0000 7f572e71e640 1 -- start start 2023-12-04T05:48:09.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f572e71e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f57280c46a0 0x7f57280c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f572e71e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f5728157180 0x7f57280c7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f572e71e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572806e660 con 0x7f57280c4a80 2023-12-04T05:48:09.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f572e71e640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5728156490 con 0x7f5728157180 2023-12-04T05:48:09.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f572e71e640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5728156cf0 con 0x7f57280c46a0 2023-12-04T05:48:09.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f572ccbb640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f57280c4a80 0x7f57280c5ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60458/0 (socket says 172.21.15.107:60458) 2023-12-04T05:48:09.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f572ccbb640 1 -- 172.21.15.107:0/2879639080 learned_addr learned my addr 172.21.15.107:0/2879639080 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:48:09.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f57277fe640 1 --2- 172.21.15.107:0/2879639080 >> v2:172.21.15.195:3300/0 conn(0x7f5728157180 0x7f57280c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5727fff640 1 --2- 172.21.15.107:0/2879639080 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f57280c46a0 0x7f57280c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5726ffd640 1 -- 172.21.15.107:0/2879639080 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736831034 0 0) 0x7f57200014a0 con 0x7f57280c4a80 2023-12-04T05:48:09.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5726ffd640 1 -- 172.21.15.107:0/2879639080 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5710003530 con 0x7f57280c4a80 2023-12-04T05:48:09.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5726ffd640 1 -- 172.21.15.107:0/2879639080 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1168819807 0 0) 0x7f5710003530 con 0x7f57280c4a80 2023-12-04T05:48:09.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5726ffd640 1 -- 172.21.15.107:0/2879639080 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f57280c46a0 msgr2=0x7f57280c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:09.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5726ffd640 1 --2- 172.21.15.107:0/2879639080 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f57280c46a0 0x7f57280c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5726ffd640 1 -- 172.21.15.107:0/2879639080 >> v2:172.21.15.195:3300/0 conn(0x7f5728157180 msgr2=0x7f57280c7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:09.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5726ffd640 1 --2- 172.21.15.107:0/2879639080 >> v2:172.21.15.195:3300/0 conn(0x7f5728157180 0x7f57280c7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5726ffd640 1 -- 172.21.15.107:0/2879639080 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572806e0a0 con 0x7f57280c4a80 2023-12-04T05:48:09.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5726ffd640 1 -- 172.21.15.107:0/2879639080 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7f5720001b90 con 0x7f57280c4a80 2023-12-04T05:48:09.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5726ffd640 1 -- 172.21.15.107:0/2879639080 >> v1:172.21.15.107:6789/0 conn(0x7f57280c4a80 legacy=0x7f57280c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:09.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5726ffd640 1 --2- 172.21.15.107:0/2879639080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5710003980 0x7f5710003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5726ffd640 1 --2- 172.21.15.107:0/2879639080 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57100047d0 0x7f5710004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5726ffd640 1 --2- 172.21.15.107:0/2879639080 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f57280c46a0 0x7f5710005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5726ffd640 1 -- 172.21.15.107:0/2879639080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5728156490 con 0x7f5710003980 2023-12-04T05:48:09.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5726ffd640 1 -- 172.21.15.107:0/2879639080 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5728156cf0 con 0x7f57100047d0 2023-12-04T05:48:09.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5726ffd640 1 -- 172.21.15.107:0/2879639080 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f571000caa0 con 0x7f57280c46a0 2023-12-04T05:48:09.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5727fff640 1 --2- 172.21.15.107:0/2879639080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5710003980 0x7f5710003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f57277fe640 1 --2- 172.21.15.107:0/2879639080 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57100047d0 0x7f5710004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f572ccbb640 1 --2- 172.21.15.107:0/2879639080 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f57280c46a0 0x7f5710005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5727fff640 1 -- 172.21.15.107:0/2879639080 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f57280c46a0 msgr2=0x7f5710005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:09.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5727fff640 1 --2- 172.21.15.107:0/2879639080 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f57280c46a0 0x7f5710005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5727fff640 1 -- 172.21.15.107:0/2879639080 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57100047d0 msgr2=0x7f5710004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:09.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5727fff640 1 --2- 172.21.15.107:0/2879639080 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57100047d0 0x7f5710004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5727fff640 1 -- 172.21.15.107:0/2879639080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5710003530 con 0x7f5710003980 2023-12-04T05:48:09.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5727fff640 1 --2- 172.21.15.107:0/2879639080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5710003980 0x7f5710003da0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f5718009ae0 tx=0x7f571801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21f5eb3d2876dcc4 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:09.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5726ffd640 1 -- 172.21.15.107:0/2879639080 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57180a9450 con 0x7f5710003980 2023-12-04T05:48:09.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5726ffd640 1 -- 172.21.15.107:0/2879639080 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57180a95b0 con 0x7f5710003980 2023-12-04T05:48:09.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.121+0000 7f5726ffd640 1 -- 172.21.15.107:0/2879639080 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57180a9960 con 0x7f5710003980 2023-12-04T05:48:09.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f572e71e640 1 -- 172.21.15.107:0/2879639080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5710003980 msgr2=0x7f5710003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:09.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f572e71e640 1 --2- 172.21.15.107:0/2879639080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5710003980 0x7f5710003da0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f5718009ae0 tx=0x7f571801ca50 comp rx=0 tx=0).stop 2023-12-04T05:48:09.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f572ccbb640 1 -- 172.21.15.107:0/2879639080 reap_dead start 2023-12-04T05:48:09.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f572e71e640 1 -- 172.21.15.107:0/2879639080 shutdown_connections 2023-12-04T05:48:09.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f572e71e640 1 -- 172.21.15.107:0/2879639080 >> 172.21.15.107:0/2879639080 conn(0x7f57280c7ff0 msgr2=0x7f57280c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:09.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f572e71e640 1 -- 172.21.15.107:0/2879639080 shutdown_connections 2023-12-04T05:48:09.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f572e71e640 1 -- 172.21.15.107:0/2879639080 wait complete. 2023-12-04T05:48:09.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f572e71e640 1 Processor -- start 2023-12-04T05:48:09.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f572e71e640 1 -- start start 2023-12-04T05:48:09.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f572e71e640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5710003980 0x7f572814c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f572e71e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57280cadf0 0x7f572814c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f572e71e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f57280c9da0 0x7f57280ca220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f572e71e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57281e3a20 con 0x7f57280cadf0 2023-12-04T05:48:09.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f572e71e640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57280ca760 con 0x7f5710003980 2023-12-04T05:48:09.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f572e71e640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f572814b270 con 0x7f57280c9da0 2023-12-04T05:48:09.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f57277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57280cadf0 0x7f572814c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f57277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57280cadf0 0x7f572814c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57612/0 (socket says 172.21.15.107:57612) 2023-12-04T05:48:09.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f57277fe640 1 -- 172.21.15.107:0/724987084 learned_addr learned my addr 172.21.15.107:0/724987084 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:48:09.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f5727fff640 1 --2- 172.21.15.107:0/724987084 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5710003980 0x7f572814c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f572ccbb640 1 --2- 172.21.15.107:0/724987084 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f57280c9da0 0x7f57280ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f57277fe640 1 -- 172.21.15.107:0/724987084 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f57280c9da0 msgr2=0x7f57280ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:09.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f57277fe640 1 --2- 172.21.15.107:0/724987084 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f57280c9da0 0x7f57280ca220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f57277fe640 1 -- 172.21.15.107:0/724987084 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5710003980 msgr2=0x7f572814c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:09.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f57277fe640 1 --2- 172.21.15.107:0/724987084 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5710003980 0x7f572814c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f57277fe640 1 -- 172.21.15.107:0/724987084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572814bdf0 con 0x7f57280cadf0 2023-12-04T05:48:09.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f572ccbb640 1 --2- 172.21.15.107:0/724987084 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f57280c9da0 0x7f57280ca220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:48:09.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f57277fe640 1 --2- 172.21.15.107:0/724987084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57280cadf0 0x7f572814c7c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f571c011660 tx=0x7f571c011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:09.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f5724ff9640 1 -- 172.21.15.107:0/724987084 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571c00f9c0 con 0x7f57280cadf0 2023-12-04T05:48:09.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f5724ff9640 1 -- 172.21.15.107:0/724987084 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571c00fb20 con 0x7f57280cadf0 2023-12-04T05:48:09.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f572e71e640 1 -- 172.21.15.107:0/724987084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57281ef140 con 0x7f57280cadf0 2023-12-04T05:48:09.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f5724ff9640 1 -- 172.21.15.107:0/724987084 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571c00fe90 con 0x7f57280cadf0 2023-12-04T05:48:09.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.125+0000 7f572e71e640 1 -- 172.21.15.107:0/724987084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f572814b7a0 con 0x7f57280cadf0 2023-12-04T05:48:09.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.129+0000 7f572e71e640 1 -- 172.21.15.107:0/724987084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f572814bf80 con 0x7f57280cadf0 2023-12-04T05:48:09.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.129+0000 7f5724ff9640 1 -- 172.21.15.107:0/724987084 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f571c0144e0 con 0x7f57280cadf0 2023-12-04T05:48:09.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.129+0000 7f5724ff9640 1 --2- 172.21.15.107:0/724987084 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5714066900 0x7f5714068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.129+0000 7f5724ff9640 1 -- 172.21.15.107:0/724987084 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f571c0868f0 con 0x7f57280cadf0 2023-12-04T05:48:09.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.129+0000 7f5727fff640 1 --2- 172.21.15.107:0/724987084 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5714066900 0x7f5714068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.129+0000 7f5727fff640 1 --2- 172.21.15.107:0/724987084 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5714066900 0x7f5714068dc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f57180ae000 tx=0x7f57180b3040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:09.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.133+0000 7f5724ff9640 1 -- 172.21.15.107:0/724987084 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f571c053830 con 0x7f57280cadf0 2023-12-04T05:48:09.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.281+0000 7f572e71e640 1 -- 172.21.15.107:0/724987084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5728067870 con 0x7f57280cadf0 2023-12-04T05:48:09.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.285+0000 7f5724ff9640 1 -- 172.21.15.107:0/724987084 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+17315 (secure 0 0 0) 0x7f571c021020 con 0x7f57280cadf0 2023-12-04T05:48:09.287 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:48:09.287 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"fsid":"603e856b-31ac-475a-8f35-21ab7faadd48","created":"2023-12-04T05:39:34.921690+0000","modified":"2023-12-04T05:45:54.798928+0000","last_up_change":"2023-12-04T05:39:46.235790+0000","last_in_change":"2023-12-04T05:39:37.587639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T05:39:47.567702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T05:39:57.644286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T05:39:58.029897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ab7111c-0ae8-4d41-b6db-e3743614322c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6817","nonce":768714279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6821","nonce":768714279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6829","nonce":768714279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":768714279},{"type":"v1","addr":"172.21.15.107:6825","nonce":768714279}]},"public_addr":"172.21.15.107:6817/768714279","cluster_addr":"172.21.15.107:6821/768714279","heartbeat_back_addr":"172.21.15.107:6829/768714279","heartbeat_front_addr":"172.21.15.107:6825/768714279","state":["exists","up"]},{"osd":1,"uuid":"baef7a1f-8026-4fab-9e70-48465271034e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6801","nonce":2496355981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6803","nonce":2496355981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6807","nonce":2496355981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2496355981},{"type":"v1","addr":"172.21.15.107:6805","nonce":2496355981}]},"public_addr":"172.21.15.107:6801/2496355981","cluster_addr":"172.21.15.107:6803/2496355981","heartbeat_back_addr":"172.21.15.107:6807/2496355981","heartbeat_front_addr":"172.21.15.107:6805/2496355981","state":["exists","up"]},{"osd":2,"uuid":"90e8cf24-7b54-484a-9d85-b44a9d3dfeab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6809","nonce":3097298224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6811","nonce":3097298224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6815","nonce":3097298224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3097298224},{"type":"v1","addr":"172.21.15.107:6813","nonce":3097298224}]},"public_addr":"172.21.15.107:6809/3097298224","cluster_addr":"172.21.15.107:6811/3097298224","heartbeat_back_addr":"172.21.15.107:6815/3097298224","heartbeat_front_addr":"172.21.15.107:6813/3097298224","state":["exists","up"]},{"osd":3,"uuid":"54658731-2b9a-4d5e-ab13-7c1be7d34e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6820","nonce":3773076671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6824","nonce":3773076671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6831","nonce":3773076671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3773076671},{"type":"v1","addr":"172.21.15.107:6828","nonce":3773076671}]},"public_addr":"172.21.15.107:6820/3773076671","cluster_addr":"172.21.15.107:6824/3773076671","heartbeat_back_addr":"172.21.15.107:6831/3773076671","heartbeat_front_addr":"172.21.15.107:6828/3773076671","state":["exists","up"]},{"osd":4,"uuid":"9a9dea13-8afe-485d-aa43-45eda691179b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6801","nonce":2747823542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6803","nonce":2747823542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6807","nonce":2747823542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2747823542},{"type":"v1","addr":"172.21.15.195:6805","nonce":2747823542}]},"public_addr":"172.21.15.195:6801/2747823542","cluster_addr":"172.21.15.195:6803/2747823542","heartbeat_back_addr":"172.21.15.195:6807/2747823542","heartbeat_front_addr":"172.21.15.195:6805/2747823542","state":["exists","up"]},{"osd":5,"uuid":"db840d83-5a7d-468a-8aa2-1997945847c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6825","nonce":1484264540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6827","nonce":1484264540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6831","nonce":1484264540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":1484264540},{"type":"v1","addr":"172.21.15.195:6829","nonce":1484264540}]},"public_addr":"172.21.15.195:6825/1484264540","cluster_addr":"172.21.15.195:6827/1484264540","heartbeat_back_addr":"172.21.15.195:6831/1484264540","heartbeat_front_addr":"172.21.15.195:6829/1484264540","state":["exists","up"]},{"osd":6,"uuid":"38b5f165-bf91-467e-a615-4b4784337689","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6811","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6813","nonce":2638471412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6815","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6817","nonce":2638471412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6823","nonce":2638471412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6819","nonce":2638471412},{"type":"v1","addr":"172.21.15.195:6821","nonce":2638471412}]},"public_addr":"172.21.15.195:6813/2638471412","cluster_addr":"172.21.15.195:6817/2638471412","heartbeat_back_addr":"172.21.15.195:6823/2638471412","heartbeat_front_addr":"172.21.15.195:6821/2638471412","state":["exists","up"]},{"osd":7,"uuid":"e5b3c596-494b-4df3-9437-bc2d52168ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6809","nonce":768189614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6812","nonce":768189614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6820","nonce":768189614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":768189614},{"type":"v1","addr":"172.21.15.195:6816","nonce":768189614}]},"public_addr":"172.21.15.195:6809/768189614","cluster_addr":"172.21.15.195:6812/768189614","heartbeat_back_addr":"172.21.15.195:6820/768189614","heartbeat_front_addr":"172.21.15.195:6816/768189614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.209077+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.086601+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.221568+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.238524+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.059477+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.148422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.079867+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T05:39:43.064916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6839/3319682203":"2023-12-05T05:45:54.798833+0000","172.21.15.107:6838/3319682203":"2023-12-05T05:45:54.798833+0000","172.21.15.195:6836/2222928136":"2023-12-05T05:45:09.028534+0000","172.21.15.195:6833/569093002":"2023-12-05T05:43:14.736286+0000","172.21.15.195:6832/569093002":"2023-12-05T05:43:14.736286+0000","172.21.15.195:6834/2222928136":"2023-12-05T05:45:09.028534+0000","172.21.15.195:6836/3769196693":"2023-12-05T05:42:14.714882+0000","172.21.15.195:6834/3769196693":"2023-12-05T05:42:14.714882+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T05:48:09.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.285+0000 7f572e71e640 1 -- 172.21.15.107:0/724987084 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5714066900 msgr2=0x7f5714068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:09.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.285+0000 7f572e71e640 1 --2- 172.21.15.107:0/724987084 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5714066900 0x7f5714068dc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f57180ae000 tx=0x7f57180b3040 comp rx=0 tx=0).stop 2023-12-04T05:48:09.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.285+0000 7f572e71e640 1 -- 172.21.15.107:0/724987084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57280cadf0 msgr2=0x7f572814c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:09.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.285+0000 7f572e71e640 1 --2- 172.21.15.107:0/724987084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57280cadf0 0x7f572814c7c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f571c011660 tx=0x7f571c011b20 comp rx=0 tx=0).stop 2023-12-04T05:48:09.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.285+0000 7f572e71e640 1 -- 172.21.15.107:0/724987084 shutdown_connections 2023-12-04T05:48:09.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.285+0000 7f572e71e640 1 --2- 172.21.15.107:0/724987084 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f5714066900 0x7f5714068dc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.285+0000 7f572e71e640 1 --2- 172.21.15.107:0/724987084 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f57280c9da0 0x7f57280ca220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.285+0000 7f572e71e640 1 --2- 172.21.15.107:0/724987084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57280cadf0 0x7f572814c7c0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.285+0000 7f572e71e640 1 --2- 172.21.15.107:0/724987084 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5710003980 0x7f572814c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.285+0000 7f572e71e640 1 -- 172.21.15.107:0/724987084 >> 172.21.15.107:0/724987084 conn(0x7f57280c7ff0 msgr2=0x7f57281e4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:09.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.285+0000 7f572e71e640 1 -- 172.21.15.107:0/724987084 shutdown_connections 2023-12-04T05:48:09.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.285+0000 7f572e71e640 1 -- 172.21.15.107:0/724987084 wait complete. 2023-12-04T05:48:09.305 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-04T05:39:47.567702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-04T05:39:57.644286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-04T05:39:58.029897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-04T05:48:09.305 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-04T05:48:09.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e9a53640 1 Processor -- start 2023-12-04T05:48:09.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e9a53640 1 -- start start 2023-12-04T05:48:09.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e9a53640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7ff3e40c56f0 0x7ff3e4158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e9a53640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e4158da0 con 0x7ff3e4072110 2023-12-04T05:48:09.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e9a53640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e41583d0 con 0x7ff3e40724f0 2023-12-04T05:48:09.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e9a53640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3e40c6170 con 0x7ff3e40c56f0 2023-12-04T05:48:09.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e2ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff3e4072110 0x7ff3e40c7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60470/0 (socket says 172.21.15.107:60470) 2023-12-04T05:48:09.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e2ffd640 1 -- 172.21.15.107:0/2373362584 learned_addr learned my addr 172.21.15.107:0/2373362584 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:48:09.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e37fe640 1 --2- 172.21.15.107:0/2373362584 >> v2:172.21.15.195:3300/0 conn(0x7ff3e40c56f0 0x7ff3e4158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e1ffb640 1 -- 172.21.15.107:0/2373362584 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106558623 0 0) 0x7ff3d4001120 con 0x7ff3e4072110 2023-12-04T05:48:09.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e1ffb640 1 -- 172.21.15.107:0/2373362584 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3cc003530 con 0x7ff3e4072110 2023-12-04T05:48:09.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e1ffb640 1 -- 172.21.15.107:0/2373362584 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1717363209 0 0) 0x7ff3cc003530 con 0x7ff3e4072110 2023-12-04T05:48:09.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e1ffb640 1 -- 172.21.15.107:0/2373362584 >> v2:172.21.15.195:3300/0 conn(0x7ff3e40c56f0 msgr2=0x7ff3e4158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:09.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e1ffb640 1 --2- 172.21.15.107:0/2373362584 >> v2:172.21.15.195:3300/0 conn(0x7ff3e40c56f0 0x7ff3e4158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e1ffb640 1 -- 172.21.15.107:0/2373362584 >> v1:172.21.15.195:6789/0 conn(0x7ff3e40724f0 legacy=0x7ff3e4140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:09.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e1ffb640 1 -- 172.21.15.107:0/2373362584 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e4157da0 con 0x7ff3e4072110 2023-12-04T05:48:09.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e1ffb640 1 -- 172.21.15.107:0/2373362584 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2639773494 0 0) 0x7ff3d4001810 con 0x7ff3e4072110 2023-12-04T05:48:09.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e1ffb640 1 -- 172.21.15.107:0/2373362584 >> v1:172.21.15.107:6789/0 conn(0x7ff3e4072110 legacy=0x7ff3e40c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:09.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e1ffb640 1 --2- 172.21.15.107:0/2373362584 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3cc003980 0x7ff3cc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e1ffb640 1 --2- 172.21.15.107:0/2373362584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3cc0047d0 0x7ff3cc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e1ffb640 1 --2- 172.21.15.107:0/2373362584 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff3cc005190 0x7ff3cc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e1ffb640 1 -- 172.21.15.107:0/2373362584 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e40c6170 con 0x7ff3cc0047d0 2023-12-04T05:48:09.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e1ffb640 1 -- 172.21.15.107:0/2373362584 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc00c040 con 0x7ff3cc003980 2023-12-04T05:48:09.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e1ffb640 1 -- 172.21.15.107:0/2373362584 --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc00c200 con 0x7ff3cc005190 2023-12-04T05:48:09.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e27fc640 1 --2- 172.21.15.107:0/2373362584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3cc0047d0 0x7ff3cc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e2ffd640 1 --2- 172.21.15.107:0/2373362584 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3cc003980 0x7ff3cc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e37fe640 1 --2- 172.21.15.107:0/2373362584 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff3cc005190 0x7ff3cc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e27fc640 1 -- 172.21.15.107:0/2373362584 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff3cc005190 msgr2=0x7ff3cc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:09.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e27fc640 1 --2- 172.21.15.107:0/2373362584 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff3cc005190 0x7ff3cc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e27fc640 1 -- 172.21.15.107:0/2373362584 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3cc003980 msgr2=0x7ff3cc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:09.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e27fc640 1 --2- 172.21.15.107:0/2373362584 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3cc003980 0x7ff3cc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e27fc640 1 -- 172.21.15.107:0/2373362584 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc003530 con 0x7ff3cc0047d0 2023-12-04T05:48:09.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e37fe640 1 --2- 172.21.15.107:0/2373362584 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff3cc005190 0x7ff3cc00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:48:09.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e2ffd640 1 --2- 172.21.15.107:0/2373362584 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3cc003980 0x7ff3cc003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T05:48:09.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.413+0000 7ff3e37fe640 1 -- 172.21.15.107:0/2373362584 reap_dead start 2023-12-04T05:48:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e27fc640 1 --2- 172.21.15.107:0/2373362584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3cc0047d0 0x7ff3cc004c50 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8001750 tx=0x7ff3d801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efcc50c2de66e97a server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e1ffb640 1 -- 172.21.15.107:0/2373362584 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d80052d0 con 0x7ff3cc0047d0 2023-12-04T05:48:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e1ffb640 1 -- 172.21.15.107:0/2373362584 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d80a7050 con 0x7ff3cc0047d0 2023-12-04T05:48:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e1ffb640 1 -- 172.21.15.107:0/2373362584 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8003600 con 0x7ff3cc0047d0 2023-12-04T05:48:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e9a53640 1 -- 172.21.15.107:0/2373362584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3cc0047d0 msgr2=0x7ff3cc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e9a53640 1 --2- 172.21.15.107:0/2373362584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3cc0047d0 0x7ff3cc004c50 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8001750 tx=0x7ff3d801c3a0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e9a53640 1 -- 172.21.15.107:0/2373362584 shutdown_connections 2023-12-04T05:48:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e9a53640 1 --2- 172.21.15.107:0/2373362584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3cc0047d0 0x7ff3cc004c50 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e9a53640 1 -- 172.21.15.107:0/2373362584 >> 172.21.15.107:0/2373362584 conn(0x7ff3e40c7ff0 msgr2=0x7ff3e40c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e9a53640 1 -- 172.21.15.107:0/2373362584 shutdown_connections 2023-12-04T05:48:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e9a53640 1 -- 172.21.15.107:0/2373362584 wait complete. 2023-12-04T05:48:09.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e9a53640 1 Processor -- start 2023-12-04T05:48:09.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e9a53640 1 -- start start 2023-12-04T05:48:09.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e9a53640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff3cc0047d0 0x7ff3e40c17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e9a53640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3e415a6b0 0x7ff3e4158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e9a53640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3e4154ed0 0x7ff3e414ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e9a53640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e41f2e20 con 0x7ff3e415a6b0 2023-12-04T05:48:09.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e9a53640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e41595f0 con 0x7ff3e4154ed0 2023-12-04T05:48:09.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e9a53640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e4152690 con 0x7ff3cc0047d0 2023-12-04T05:48:09.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3e415a6b0 0x7ff3e4158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3e415a6b0 0x7ff3e4158f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57634/0 (socket says 172.21.15.107:57634) 2023-12-04T05:48:09.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e27fc640 1 -- 172.21.15.107:0/3559020507 learned_addr learned my addr 172.21.15.107:0/3559020507 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:48:09.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e2ffd640 1 --2- 172.21.15.107:0/3559020507 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff3cc0047d0 0x7ff3e40c17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e37fe640 1 --2- 172.21.15.107:0/3559020507 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3e4154ed0 0x7ff3e414ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e27fc640 1 -- 172.21.15.107:0/3559020507 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff3cc0047d0 msgr2=0x7ff3e40c17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:09.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e27fc640 1 --2- 172.21.15.107:0/3559020507 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff3cc0047d0 0x7ff3e40c17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e27fc640 1 -- 172.21.15.107:0/3559020507 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3e4154ed0 msgr2=0x7ff3e414ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:09.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e27fc640 1 --2- 172.21.15.107:0/3559020507 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3e4154ed0 0x7ff3e414ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.417+0000 7ff3e27fc640 1 -- 172.21.15.107:0/3559020507 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e4152970 con 0x7ff3e415a6b0 2023-12-04T05:48:09.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.421+0000 7ff3e27fc640 1 --2- 172.21.15.107:0/3559020507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3e415a6b0 0x7ff3e4158f60 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ff3d801c1d0 tx=0x7ff3cc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:09.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.421+0000 7ff3cbfff640 1 -- 172.21.15.107:0/3559020507 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8007440 con 0x7ff3e415a6b0 2023-12-04T05:48:09.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.421+0000 7ff3cbfff640 1 -- 172.21.15.107:0/3559020507 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d80075a0 con 0x7ff3e415a6b0 2023-12-04T05:48:09.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.421+0000 7ff3e9a53640 1 -- 172.21.15.107:0/3559020507 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e4152c50 con 0x7ff3e415a6b0 2023-12-04T05:48:09.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.421+0000 7ff3cbfff640 1 -- 172.21.15.107:0/3559020507 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8006a20 con 0x7ff3e415a6b0 2023-12-04T05:48:09.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.421+0000 7ff3e9a53640 1 -- 172.21.15.107:0/3559020507 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e4070ff0 con 0x7ff3e415a6b0 2023-12-04T05:48:09.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.421+0000 7ff3e9a53640 1 -- 172.21.15.107:0/3559020507 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e4153100 con 0x7ff3e415a6b0 2023-12-04T05:48:09.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.421+0000 7ff3cbfff640 1 -- 172.21.15.107:0/3559020507 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7ff3d8006b80 con 0x7ff3e415a6b0 2023-12-04T05:48:09.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.421+0000 7ff3cbfff640 1 --2- 172.21.15.107:0/3559020507 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff3d0066900 0x7ff3d0068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.421+0000 7ff3e2ffd640 1 --2- 172.21.15.107:0/3559020507 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff3d0066900 0x7ff3d0068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.421+0000 7ff3cbfff640 1 -- 172.21.15.107:0/3559020507 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7ff3d811e7c0 con 0x7ff3e415a6b0 2023-12-04T05:48:09.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.421+0000 7ff3e2ffd640 1 --2- 172.21.15.107:0/3559020507 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff3d0066900 0x7ff3d0068dc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff3d40059a0 tx=0x7ff3d4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:09.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.425+0000 7ff3cbfff640 1 -- 172.21.15.107:0/3559020507 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff3d80eb7a0 con 0x7ff3e415a6b0 2023-12-04T05:48:09.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.569+0000 7ff3e9a53640 1 -- 172.21.15.107:0/3559020507 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff3e4067870 con 0x7ff3e415a6b0 2023-12-04T05:48:09.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.569+0000 7ff3cbfff640 1 -- 172.21.15.107:0/3559020507 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff3d80eedc0 con 0x7ff3e415a6b0 2023-12-04T05:48:09.572 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-12-04T05:48:09.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.573+0000 7ff3e9a53640 1 -- 172.21.15.107:0/3559020507 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff3d0066900 msgr2=0x7ff3d0068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:09.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.573+0000 7ff3e9a53640 1 --2- 172.21.15.107:0/3559020507 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff3d0066900 0x7ff3d0068dc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff3d40059a0 tx=0x7ff3d4009040 comp rx=0 tx=0).stop 2023-12-04T05:48:09.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.573+0000 7ff3e9a53640 1 -- 172.21.15.107:0/3559020507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3e415a6b0 msgr2=0x7ff3e4158f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:09.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.573+0000 7ff3e9a53640 1 --2- 172.21.15.107:0/3559020507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3e415a6b0 0x7ff3e4158f60 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ff3d801c1d0 tx=0x7ff3cc005800 comp rx=0 tx=0).stop 2023-12-04T05:48:09.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.573+0000 7ff3e9a53640 1 -- 172.21.15.107:0/3559020507 shutdown_connections 2023-12-04T05:48:09.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.573+0000 7ff3e9a53640 1 --2- 172.21.15.107:0/3559020507 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7ff3d0066900 0x7ff3d0068dc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.573+0000 7ff3e9a53640 1 --2- 172.21.15.107:0/3559020507 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3e4154ed0 0x7ff3e414ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.573+0000 7ff3e9a53640 1 --2- 172.21.15.107:0/3559020507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3e415a6b0 0x7ff3e4158f60 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.573+0000 7ff3e9a53640 1 --2- 172.21.15.107:0/3559020507 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7ff3cc0047d0 0x7ff3e40c17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.573+0000 7ff3e9a53640 1 -- 172.21.15.107:0/3559020507 >> 172.21.15.107:0/3559020507 conn(0x7ff3e40c7ff0 msgr2=0x7ff3e40c71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:09.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.573+0000 7ff3e9a53640 1 -- 172.21.15.107:0/3559020507 shutdown_connections 2023-12-04T05:48:09.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.573+0000 7ff3e9a53640 1 -- 172.21.15.107:0/3559020507 wait complete. 2023-12-04T05:48:09.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-04T05:48:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.713+0000 7fe128ef7640 1 Processor -- start 2023-12-04T05:48:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.713+0000 7fe128ef7640 1 -- start start 2023-12-04T05:48:09.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.713+0000 7fe128ef7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fe124151180 0x7fe1240c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.713+0000 7fe128ef7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe124070380 0x7fe124070780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.713+0000 7fe128ef7640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1240700c0 con 0x7fe1240714f0 2023-12-04T05:48:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.713+0000 7fe128ef7640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe124156df0 con 0x7fe124070380 2023-12-04T05:48:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.713+0000 7fe128ef7640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1241570d0 con 0x7fe124151180 2023-12-04T05:48:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.713+0000 7fe122d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe124070380 0x7fe124070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.713+0000 7fe122d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe124070380 0x7fe124070780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57642/0 (socket says 172.21.15.107:57642) 2023-12-04T05:48:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.713+0000 7fe122d9d640 1 -- 172.21.15.107:0/2630549196 learned_addr learned my addr 172.21.15.107:0/2630549196 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:48:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.713+0000 7fe12259c640 1 --2- 172.21.15.107:0/2630549196 >> v2:172.21.15.195:3300/0 conn(0x7fe124151180 0x7fe1240c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.713+0000 7fe122d9d640 1 -- 172.21.15.107:0/2630549196 >> v2:172.21.15.195:3300/0 conn(0x7fe124151180 msgr2=0x7fe1240c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.713+0000 7fe122d9d640 1 --2- 172.21.15.107:0/2630549196 >> v2:172.21.15.195:3300/0 conn(0x7fe124151180 0x7fe1240c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.713+0000 7fe122d9d640 1 -- 172.21.15.107:0/2630549196 >> v1:172.21.15.195:6789/0 conn(0x7fe1240714f0 legacy=0x7fe124071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:09.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.713+0000 7fe122d9d640 1 -- 172.21.15.107:0/2630549196 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1241573b0 con 0x7fe124070380 2023-12-04T05:48:09.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.713+0000 7fe122d9d640 1 --2- 172.21.15.107:0/2630549196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe124070380 0x7fe124070780 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fe12406ba20 tx=0x7fe11c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53dc8a9f18b0cb94 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:09.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe121d9b640 1 -- 172.21.15.107:0/2630549196 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11c0af400 con 0x7fe124070380 2023-12-04T05:48:09.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe121d9b640 1 -- 172.21.15.107:0/2630549196 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe11c0ab050 con 0x7fe124070380 2023-12-04T05:48:09.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe121d9b640 1 -- 172.21.15.107:0/2630549196 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11c0af780 con 0x7fe124070380 2023-12-04T05:48:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe128ef7640 1 -- 172.21.15.107:0/2630549196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe124070380 msgr2=0x7fe124070780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe128ef7640 1 --2- 172.21.15.107:0/2630549196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe124070380 0x7fe124070780 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fe12406ba20 tx=0x7fe11c01f1c0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe128ef7640 1 -- 172.21.15.107:0/2630549196 shutdown_connections 2023-12-04T05:48:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe128ef7640 1 --2- 172.21.15.107:0/2630549196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe124070380 0x7fe124070780 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe128ef7640 1 --2- 172.21.15.107:0/2630549196 >> v2:172.21.15.195:3300/0 conn(0x7fe124151180 0x7fe1240c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe128ef7640 1 -- 172.21.15.107:0/2630549196 >> 172.21.15.107:0/2630549196 conn(0x7fe1240c7ff0 msgr2=0x7fe1240c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe128ef7640 1 -- 172.21.15.107:0/2630549196 shutdown_connections 2023-12-04T05:48:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe128ef7640 1 -- 172.21.15.107:0/2630549196 wait complete. 2023-12-04T05:48:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe128ef7640 1 Processor -- start 2023-12-04T05:48:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe128ef7640 1 -- start start 2023-12-04T05:48:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe128ef7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe124070380 0x7fe1241e39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe128ef7640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1240714f0 0x7fe1240c4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe128ef7640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe124151180 0x7fe124146d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe128ef7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1241ed4f0 con 0x7fe124070380 2023-12-04T05:48:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe128ef7640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1241ec960 con 0x7fe1240714f0 2023-12-04T05:48:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe128ef7640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1241ed910 con 0x7fe124151180 2023-12-04T05:48:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe12259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe124070380 0x7fe1241e39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe12259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe124070380 0x7fe1241e39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57652/0 (socket says 172.21.15.107:57652) 2023-12-04T05:48:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe12259c640 1 -- 172.21.15.107:0/1504356147 learned_addr learned my addr 172.21.15.107:0/1504356147 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:48:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe122d9d640 1 --2- 172.21.15.107:0/1504356147 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe124151180 0x7fe124146d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe12259c640 1 -- 172.21.15.107:0/1504356147 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe124151180 msgr2=0x7fe124146d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe12259c640 1 --2- 172.21.15.107:0/1504356147 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe124151180 0x7fe124146d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe12259c640 1 -- 172.21.15.107:0/1504356147 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1240714f0 msgr2=0x7fe1240c4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe12259c640 1 --2- 172.21.15.107:0/1504356147 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1240714f0 0x7fe1240c4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe12259c640 1 -- 172.21.15.107:0/1504356147 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11c009b30 con 0x7fe124070380 2023-12-04T05:48:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.717+0000 7fe12259c640 1 --2- 172.21.15.107:0/1504356147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe124070380 0x7fe1241e39d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fe11400fa40 tx=0x7fe11400ff00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.721+0000 7fe11b7fe640 1 -- 172.21.15.107:0/1504356147 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe114007c90 con 0x7fe124070380 2023-12-04T05:48:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.721+0000 7fe11b7fe640 1 -- 172.21.15.107:0/1504356147 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe114007df0 con 0x7fe124070380 2023-12-04T05:48:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.721+0000 7fe128ef7640 1 -- 172.21.15.107:0/1504356147 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1241effe0 con 0x7fe124070380 2023-12-04T05:48:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.721+0000 7fe11b7fe640 1 -- 172.21.15.107:0/1504356147 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe114011670 con 0x7fe124070380 2023-12-04T05:48:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.721+0000 7fe128ef7640 1 -- 172.21.15.107:0/1504356147 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1241ece30 con 0x7fe124070380 2023-12-04T05:48:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.721+0000 7fe128ef7640 1 -- 172.21.15.107:0/1504356147 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe124067870 con 0x7fe124070380 2023-12-04T05:48:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.721+0000 7fe11b7fe640 1 -- 172.21.15.107:0/1504356147 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fe1140117d0 con 0x7fe124070380 2023-12-04T05:48:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.721+0000 7fe11b7fe640 1 --2- 172.21.15.107:0/1504356147 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe104066900 0x7fe104068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.721+0000 7fe11b7fe640 1 -- 172.21.15.107:0/1504356147 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fe1140869c0 con 0x7fe124070380 2023-12-04T05:48:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.721+0000 7fe119d9b640 1 --2- 172.21.15.107:0/1504356147 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe104066900 0x7fe104068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.721+0000 7fe119d9b640 1 --2- 172.21.15.107:0/1504356147 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe104066900 0x7fe104068dc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe10c000f70 tx=0x7fe10c0079c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:09.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.725+0000 7fe11b7fe640 1 -- 172.21.15.107:0/1504356147 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe114053a30 con 0x7fe124070380 2023-12-04T05:48:09.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:48:09.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.877+0000 7f48aae51640 1 Processor -- start 2023-12-04T05:48:09.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.877+0000 7f48aae51640 1 -- start start 2023-12-04T05:48:09.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.877+0000 7f48aae51640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f48a40ca2e0 0x7f48a40c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.877+0000 7f48aae51640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48a40ca6c0 0x7f48a40c7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.877+0000 7f48aae51640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f48a4151990 0x7f48a4151db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.877+0000 7f48aae51640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a4158b80 con 0x7f48a4151990 2023-12-04T05:48:09.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.877+0000 7f48aae51640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a4157f70 con 0x7f48a40ca2e0 2023-12-04T05:48:09.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.877+0000 7f48aae51640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a4158250 con 0x7f48a40ca6c0 2023-12-04T05:48:09.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.877+0000 7f48a93ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f48a4151990 0x7f48a4151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.877+0000 7f48a93ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f48a4151990 0x7f48a4151db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57660/0 (socket says 172.21.15.107:57660) 2023-12-04T05:48:09.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.877+0000 7f48a93ee640 1 -- 172.21.15.107:0/3498339280 learned_addr learned my addr 172.21.15.107:0/3498339280 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:48:09.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.877+0000 7f48a8bed640 1 --2- 172.21.15.107:0/3498339280 >> v2:172.21.15.195:3300/0 conn(0x7f48a40ca2e0 0x7f48a40c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.877+0000 7f48a3fff640 1 --2- 172.21.15.107:0/3498339280 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48a40ca6c0 0x7f48a40c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.877+0000 7f48a93ee640 1 -- 172.21.15.107:0/3498339280 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48a40ca6c0 msgr2=0x7f48a40c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:09.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.877+0000 7f48a93ee640 1 --2- 172.21.15.107:0/3498339280 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48a40ca6c0 0x7f48a40c7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.877+0000 7f48a93ee640 1 -- 172.21.15.107:0/3498339280 >> v2:172.21.15.195:3300/0 conn(0x7f48a40ca2e0 msgr2=0x7f48a40c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:09.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.877+0000 7f48a93ee640 1 --2- 172.21.15.107:0/3498339280 >> v2:172.21.15.195:3300/0 conn(0x7f48a40ca2e0 0x7f48a40c6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.877+0000 7f48a93ee640 1 -- 172.21.15.107:0/3498339280 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a4158530 con 0x7f48a4151990 2023-12-04T05:48:09.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.877+0000 7fe128ef7640 1 -- 172.21.15.107:0/1504356147 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe12414f920 con 0x7fe124070380 2023-12-04T05:48:09.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.877+0000 7fe11b7fe640 1 -- 172.21.15.107:0/1504356147 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe11400d550 con 0x7fe124070380 2023-12-04T05:48:09.883 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-12-04T05:48:09.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7f48a93ee640 1 --2- 172.21.15.107:0/3498339280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48a4151990 0x7f48a4151db0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f48a4070720 tx=0x7f489c01c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4191a39c7afb6a2 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:09.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7f48a37fe640 1 -- 172.21.15.107:0/3498339280 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c0a5810 con 0x7f48a4151990 2023-12-04T05:48:09.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7f48a37fe640 1 -- 172.21.15.107:0/3498339280 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f489c0a5970 con 0x7f48a4151990 2023-12-04T05:48:09.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7f48a37fe640 1 -- 172.21.15.107:0/3498339280 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c0a5d00 con 0x7f48a4151990 2023-12-04T05:48:09.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7fe128ef7640 1 -- 172.21.15.107:0/1504356147 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe104066900 msgr2=0x7fe104068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:09.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7fe128ef7640 1 --2- 172.21.15.107:0/1504356147 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe104066900 0x7fe104068dc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe10c000f70 tx=0x7fe10c0079c0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7fe128ef7640 1 -- 172.21.15.107:0/1504356147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe124070380 msgr2=0x7fe1241e39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:09.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7fe128ef7640 1 --2- 172.21.15.107:0/1504356147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe124070380 0x7fe1241e39d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fe11400fa40 tx=0x7fe11400ff00 comp rx=0 tx=0).stop 2023-12-04T05:48:09.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7fe128ef7640 1 -- 172.21.15.107:0/1504356147 shutdown_connections 2023-12-04T05:48:09.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7fe128ef7640 1 --2- 172.21.15.107:0/1504356147 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fe104066900 0x7fe104068dc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7fe128ef7640 1 --2- 172.21.15.107:0/1504356147 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fe124151180 0x7fe124146d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7fe128ef7640 1 --2- 172.21.15.107:0/1504356147 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe1240714f0 0x7fe1240c4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7fe128ef7640 1 --2- 172.21.15.107:0/1504356147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe124070380 0x7fe1241e39d0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7fe128ef7640 1 -- 172.21.15.107:0/1504356147 >> 172.21.15.107:0/1504356147 conn(0x7fe1240c7ff0 msgr2=0x7fe12414b430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:09.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7fe128ef7640 1 -- 172.21.15.107:0/1504356147 shutdown_connections 2023-12-04T05:48:09.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7fe128ef7640 1 -- 172.21.15.107:0/1504356147 wait complete. 2023-12-04T05:48:09.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7f48aae51640 1 -- 172.21.15.107:0/3498339280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48a4151990 msgr2=0x7f48a4151db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:09.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7f48aae51640 1 --2- 172.21.15.107:0/3498339280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48a4151990 0x7f48a4151db0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f48a4070720 tx=0x7f489c01c9d0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7f48aae51640 1 -- 172.21.15.107:0/3498339280 shutdown_connections 2023-12-04T05:48:09.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7f48aae51640 1 --2- 172.21.15.107:0/3498339280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48a4151990 0x7f48a4151db0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7f48aae51640 1 --2- 172.21.15.107:0/3498339280 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48a40ca6c0 0x7f48a40c7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7f48aae51640 1 --2- 172.21.15.107:0/3498339280 >> v2:172.21.15.195:3300/0 conn(0x7f48a40ca2e0 0x7f48a40c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7f48aae51640 1 -- 172.21.15.107:0/3498339280 >> 172.21.15.107:0/3498339280 conn(0x7f48a40663a0 msgr2=0x7f48a40667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:09.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7f48aae51640 1 -- 172.21.15.107:0/3498339280 shutdown_connections 2023-12-04T05:48:09.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7f48aae51640 1 -- 172.21.15.107:0/3498339280 wait complete. 2023-12-04T05:48:09.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7f48aae51640 1 Processor -- start 2023-12-04T05:48:09.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7f48aae51640 1 -- start start 2023-12-04T05:48:09.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7f48aae51640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48a40ca2e0 0x7f48a40cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7f48aae51640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48a40ca6c0 0x7f48a4150030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7f48aae51640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48a4151990 0x7f48a4150570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7f48aae51640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a41f4b90 con 0x7f48a4151990 2023-12-04T05:48:09.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7f48aae51640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a414fba0 con 0x7f48a40ca2e0 2023-12-04T05:48:09.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.881+0000 7f48aae51640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a41f0980 con 0x7f48a40ca6c0 2023-12-04T05:48:09.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.885+0000 7f48a93ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48a4151990 0x7f48a4150570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.885+0000 7f48a93ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48a4151990 0x7f48a4150570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57662/0 (socket says 172.21.15.107:57662) 2023-12-04T05:48:09.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.885+0000 7f48a93ee640 1 -- 172.21.15.107:0/1551031681 learned_addr learned my addr 172.21.15.107:0/1551031681 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:48:09.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.885+0000 7f48a3fff640 1 --2- 172.21.15.107:0/1551031681 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48a40ca6c0 0x7f48a4150030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.885+0000 7f48a8bed640 1 --2- 172.21.15.107:0/1551031681 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48a40ca2e0 0x7f48a40cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.885+0000 7f48a93ee640 1 -- 172.21.15.107:0/1551031681 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48a40ca6c0 msgr2=0x7f48a4150030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:09.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.885+0000 7f48a93ee640 1 --2- 172.21.15.107:0/1551031681 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48a40ca6c0 0x7f48a4150030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.885+0000 7f48a93ee640 1 -- 172.21.15.107:0/1551031681 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48a40ca2e0 msgr2=0x7f48a40cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:09.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.885+0000 7f48a93ee640 1 --2- 172.21.15.107:0/1551031681 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48a40ca2e0 0x7f48a40cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:09.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.885+0000 7f48a93ee640 1 -- 172.21.15.107:0/1551031681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489c009b30 con 0x7f48a4151990 2023-12-04T05:48:09.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.885+0000 7f48a93ee640 1 --2- 172.21.15.107:0/1551031681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48a4151990 0x7f48a4150570 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f489c0af040 tx=0x7f489c019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:09.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.885+0000 7f48a17fa640 1 -- 172.21.15.107:0/1551031681 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c0be070 con 0x7f48a4151990 2023-12-04T05:48:09.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.885+0000 7f48a17fa640 1 -- 172.21.15.107:0/1551031681 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f489c0b8440 con 0x7f48a4151990 2023-12-04T05:48:09.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.885+0000 7f48aae51640 1 -- 172.21.15.107:0/1551031681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a41f0c40 con 0x7f48a4151990 2023-12-04T05:48:09.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.885+0000 7f48a17fa640 1 -- 172.21.15.107:0/1551031681 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c0b87f0 con 0x7f48a4151990 2023-12-04T05:48:09.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.885+0000 7f48aae51640 1 -- 172.21.15.107:0/1551031681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a41f1200 con 0x7f48a4151990 2023-12-04T05:48:09.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.885+0000 7f48aae51640 1 -- 172.21.15.107:0/1551031681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a4067ec0 con 0x7f48a4151990 2023-12-04T05:48:09.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.885+0000 7f48a17fa640 1 -- 172.21.15.107:0/1551031681 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f489c0b8950 con 0x7f48a4151990 2023-12-04T05:48:09.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.889+0000 7f48a17fa640 1 --2- 172.21.15.107:0/1551031681 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4888066900 0x7f4888068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:09.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.889+0000 7f48a17fa640 1 -- 172.21.15.107:0/1551031681 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f489c11fd60 con 0x7f48a4151990 2023-12-04T05:48:09.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.889+0000 7f48a8bed640 1 --2- 172.21.15.107:0/1551031681 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4888066900 0x7f4888068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:09.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.889+0000 7f48a8bed640 1 --2- 172.21.15.107:0/1551031681 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4888066900 0x7f4888068dc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f489800ba30 tx=0x7f4898008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:09.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:09.889+0000 7f48a17fa640 1 -- 172.21.15.107:0/1551031681 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f489c0f08d0 con 0x7f48a4151990 2023-12-04T05:48:09.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-04T05:48:10.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.005+0000 7f3280bc9640 1 Processor -- start 2023-12-04T05:48:10.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.005+0000 7f3280bc9640 1 -- start start 2023-12-04T05:48:10.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.005+0000 7f3280bc9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f327c06c1e0 0x7f327c06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:10.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.005+0000 7f3280bc9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327c159ee0 con 0x7f327c15d670 2023-12-04T05:48:10.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.005+0000 7f3280bc9640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327c15a1a0 con 0x7f327c1433e0 2023-12-04T05:48:10.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.005+0000 7f3280bc9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f327c072910 con 0x7f327c06c1e0 2023-12-04T05:48:10.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.005+0000 7f3279d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f327c06c1e0 0x7f327c06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:10.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.005+0000 7f327ad9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f327c15d670 0x7f327c15da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60478/0 (socket says 172.21.15.107:60478) 2023-12-04T05:48:10.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.005+0000 7f3279d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f327c06c1e0 0x7f327c06ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57664/0 (socket says 172.21.15.107:57664) 2023-12-04T05:48:10.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.005+0000 7f327ad9d640 1 -- 172.21.15.107:0/500537876 learned_addr learned my addr 172.21.15.107:0/500537876 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-12-04T05:48:10.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.005+0000 7f3279d9b640 1 -- 172.21.15.107:0/500537876 >> v1:172.21.15.195:6789/0 conn(0x7f327c1433e0 legacy=0x7f327c06a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:10.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.005+0000 7f3279d9b640 1 -- 172.21.15.107:0/500537876 >> v1:172.21.15.107:6789/0 conn(0x7f327c15d670 legacy=0x7f327c15da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:10.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.005+0000 7f3279d9b640 1 -- 172.21.15.107:0/500537876 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327c15a480 con 0x7f327c06c1e0 2023-12-04T05:48:10.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.005+0000 7f327959a640 1 -- 172.21.15.107:0/500537876 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499851972 0 0) 0x7f327c159ee0 con 0x7f327c15d670 2023-12-04T05:48:10.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f3279d9b640 1 --2- 172.21.15.107:0/500537876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f327c06c1e0 0x7f327c06ed10 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f326c0097c0 tx=0x7f326c01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3b5270625581f42 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:10.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f327959a640 1 -- 172.21.15.107:0/500537876 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c0ac620 con 0x7f327c06c1e0 2023-12-04T05:48:10.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f327959a640 1 -- 172.21.15.107:0/500537876 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326c0ac780 con 0x7f327c06c1e0 2023-12-04T05:48:10.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f327959a640 1 -- 172.21.15.107:0/500537876 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c0acb10 con 0x7f327c06c1e0 2023-12-04T05:48:10.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f3280bc9640 1 -- 172.21.15.107:0/500537876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f327c06c1e0 msgr2=0x7f327c06ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:10.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f3280bc9640 1 --2- 172.21.15.107:0/500537876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f327c06c1e0 0x7f327c06ed10 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f326c0097c0 tx=0x7f326c01c3c0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f3280bc9640 1 -- 172.21.15.107:0/500537876 shutdown_connections 2023-12-04T05:48:10.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f3280bc9640 1 --2- 172.21.15.107:0/500537876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f327c06c1e0 0x7f327c06ed10 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f3280bc9640 1 -- 172.21.15.107:0/500537876 >> 172.21.15.107:0/500537876 conn(0x7f327c06b6c0 msgr2=0x7f327c06bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:10.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f3280bc9640 1 -- 172.21.15.107:0/500537876 shutdown_connections 2023-12-04T05:48:10.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f3280bc9640 1 -- 172.21.15.107:0/500537876 wait complete. 2023-12-04T05:48:10.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f3280bc9640 1 Processor -- start 2023-12-04T05:48:10.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f3280bc9640 1 -- start start 2023-12-04T05:48:10.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f3280bc9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f327c06c1e0 0x7f327c13b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:10.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f3280bc9640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f327c1433e0 0x7f327c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:10.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f3280bc9640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f327c15d670 0x7f327c13a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:10.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f3280bc9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327c13b620 con 0x7f327c06c1e0 2023-12-04T05:48:10.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f3280bc9640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327c157830 con 0x7f327c15d670 2023-12-04T05:48:10.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f3280bc9640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f327c1f3190 con 0x7f327c1433e0 2023-12-04T05:48:10.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f327a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f327c06c1e0 0x7f327c13b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:10.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f327a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f327c06c1e0 0x7f327c13b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57668/0 (socket says 172.21.15.107:57668) 2023-12-04T05:48:10.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f327a59c640 1 -- 172.21.15.107:0/1046518349 learned_addr learned my addr 172.21.15.107:0/1046518349 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:48:10.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f3279d9b640 1 --2- 172.21.15.107:0/1046518349 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f327c1433e0 0x7f327c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:10.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f327a59c640 1 -- 172.21.15.107:0/1046518349 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f327c1433e0 msgr2=0x7f327c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:10.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f327ad9d640 1 --2- 172.21.15.107:0/1046518349 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f327c15d670 0x7f327c13a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:10.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f327a59c640 1 --2- 172.21.15.107:0/1046518349 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f327c1433e0 0x7f327c158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f327a59c640 1 -- 172.21.15.107:0/1046518349 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f327c15d670 msgr2=0x7f327c13a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:10.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f327a59c640 1 --2- 172.21.15.107:0/1046518349 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f327c15d670 0x7f327c13a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f327a59c640 1 -- 172.21.15.107:0/1046518349 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326c009420 con 0x7f327c06c1e0 2023-12-04T05:48:10.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.009+0000 7f327a59c640 1 --2- 172.21.15.107:0/1046518349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f327c06c1e0 0x7f327c13b0e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f327000c500 tx=0x7f327000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:10.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.013+0000 7f326affd640 1 -- 172.21.15.107:0/1046518349 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3270005220 con 0x7f327c06c1e0 2023-12-04T05:48:10.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.013+0000 7f326affd640 1 -- 172.21.15.107:0/1046518349 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3270005380 con 0x7f327c06c1e0 2023-12-04T05:48:10.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.013+0000 7f3280bc9640 1 -- 172.21.15.107:0/1046518349 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f327c1f3470 con 0x7f327c06c1e0 2023-12-04T05:48:10.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.013+0000 7f326affd640 1 -- 172.21.15.107:0/1046518349 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3270007270 con 0x7f327c06c1e0 2023-12-04T05:48:10.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.013+0000 7f3280bc9640 1 -- 172.21.15.107:0/1046518349 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f327c157310 con 0x7f327c06c1e0 2023-12-04T05:48:10.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.013+0000 7f3280bc9640 1 -- 172.21.15.107:0/1046518349 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f327c067ec0 con 0x7f327c06c1e0 2023-12-04T05:48:10.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.013+0000 7f326affd640 1 -- 172.21.15.107:0/1046518349 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7f32700073d0 con 0x7f327c06c1e0 2023-12-04T05:48:10.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.013+0000 7f326affd640 1 --2- 172.21.15.107:0/1046518349 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f325c0668d0 0x7f325c068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:10.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.013+0000 7f3279d9b640 1 --2- 172.21.15.107:0/1046518349 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f325c0668d0 0x7f325c068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:10.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.013+0000 7f326affd640 1 -- 172.21.15.107:0/1046518349 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f3270083ef0 con 0x7f327c06c1e0 2023-12-04T05:48:10.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.013+0000 7f3279d9b640 1 --2- 172.21.15.107:0/1046518349 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f325c0668d0 0x7f325c068d90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f326c0ae000 tx=0x7f326c0a7000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:10.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.017+0000 7f326affd640 1 -- 172.21.15.107:0/1046518349 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3270050f70 con 0x7f327c06c1e0 2023-12-04T05:48:10.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.069+0000 7f48aae51640 1 -- 172.21.15.107:0/1551031681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48a414fce0 con 0x7f48a4151990 2023-12-04T05:48:10.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.073+0000 7f48a17fa640 1 -- 172.21.15.107:0/1551031681 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f489c0f0460 con 0x7f48a4151990 2023-12-04T05:48:10.074 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-04T05:48:10.075 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2752497349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6835","nonce":2752497349},{"type":"v1","addr":"172.21.15.107:6837","nonce":2752497349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T05:39:59.032019+0000","modified":"2023-12-04T05:46:11.040247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4288,"mds_1":5190},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.195:6837/3412865477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6835","nonce":3412865477},{"type":"v1","addr":"172.21.15.195:6837","nonce":3412865477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5190":{"gid":5190,"name":"e","rank":1,"incarnation":45,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6836/2928623624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2928623624},{"type":"v1","addr":"172.21.15.107:6836","nonce":2928623624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T05:48:10.075 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-12-04T05:48:10.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.073+0000 7f48aae51640 1 -- 172.21.15.107:0/1551031681 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4888066900 msgr2=0x7f4888068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:10.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.073+0000 7f48aae51640 1 --2- 172.21.15.107:0/1551031681 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4888066900 0x7f4888068dc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f489800ba30 tx=0x7f4898008040 comp rx=0 tx=0).stop 2023-12-04T05:48:10.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.073+0000 7f48aae51640 1 -- 172.21.15.107:0/1551031681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48a4151990 msgr2=0x7f48a4150570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:10.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.073+0000 7f48aae51640 1 --2- 172.21.15.107:0/1551031681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48a4151990 0x7f48a4150570 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f489c0af040 tx=0x7f489c019770 comp rx=0 tx=0).stop 2023-12-04T05:48:10.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.073+0000 7f48aae51640 1 -- 172.21.15.107:0/1551031681 shutdown_connections 2023-12-04T05:48:10.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.073+0000 7f48aae51640 1 --2- 172.21.15.107:0/1551031681 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f4888066900 0x7f4888068dc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.073+0000 7f48aae51640 1 --2- 172.21.15.107:0/1551031681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48a4151990 0x7f48a4150570 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.073+0000 7f48aae51640 1 --2- 172.21.15.107:0/1551031681 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f48a40ca6c0 0x7f48a4150030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.073+0000 7f48aae51640 1 --2- 172.21.15.107:0/1551031681 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48a40ca2e0 0x7f48a40cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.073+0000 7f48aae51640 1 -- 172.21.15.107:0/1551031681 >> 172.21.15.107:0/1551031681 conn(0x7f48a40663a0 msgr2=0x7f48a40c9090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:10.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.073+0000 7f48aae51640 1 -- 172.21.15.107:0/1551031681 shutdown_connections 2023-12-04T05:48:10.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.073+0000 7f48aae51640 1 -- 172.21.15.107:0/1551031681 wait complete. 2023-12-04T05:48:10.093 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-12-04T05:48:10.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.165+0000 7f3280bc9640 1 -- 172.21.15.107:0/1046518349 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f327c0c8860 con 0x7f327c06c1e0 2023-12-04T05:48:10.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.165+0000 7f326affd640 1 -- 172.21.15.107:0/1046518349 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3270054590 con 0x7f327c06c1e0 2023-12-04T05:48:10.168 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-12-04T05:48:10.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.165+0000 7f3280bc9640 1 -- 172.21.15.107:0/1046518349 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f325c0668d0 msgr2=0x7f325c068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:10.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.165+0000 7f3280bc9640 1 --2- 172.21.15.107:0/1046518349 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f325c0668d0 0x7f325c068d90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f326c0ae000 tx=0x7f326c0a7000 comp rx=0 tx=0).stop 2023-12-04T05:48:10.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.165+0000 7f3280bc9640 1 -- 172.21.15.107:0/1046518349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f327c06c1e0 msgr2=0x7f327c13b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:10.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.165+0000 7f3280bc9640 1 --2- 172.21.15.107:0/1046518349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f327c06c1e0 0x7f327c13b0e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f327000c500 tx=0x7f327000c9c0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.165+0000 7f3280bc9640 1 -- 172.21.15.107:0/1046518349 shutdown_connections 2023-12-04T05:48:10.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.165+0000 7f3280bc9640 1 --2- 172.21.15.107:0/1046518349 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7f325c0668d0 0x7f325c068d90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.165+0000 7f3280bc9640 1 --2- 172.21.15.107:0/1046518349 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f327c15d670 0x7f327c13a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.165+0000 7f3280bc9640 1 --2- 172.21.15.107:0/1046518349 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f327c1433e0 0x7f327c158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.165+0000 7f3280bc9640 1 --2- 172.21.15.107:0/1046518349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f327c06c1e0 0x7f327c13b0e0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.165+0000 7f3280bc9640 1 -- 172.21.15.107:0/1046518349 >> 172.21.15.107:0/1046518349 conn(0x7f327c06b6c0 msgr2=0x7f327c06baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:10.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.165+0000 7f3280bc9640 1 -- 172.21.15.107:0/1046518349 shutdown_connections 2023-12-04T05:48:10.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.165+0000 7f3280bc9640 1 -- 172.21.15.107:0/1046518349 wait complete. 2023-12-04T05:48:10.185 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-04T05:48:10.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.293+0000 7fb698f1c640 1 Processor -- start 2023-12-04T05:48:10.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.293+0000 7fb698f1c640 1 -- start start 2023-12-04T05:48:10.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.293+0000 7fb698f1c640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fb69406e730 0x7fb694070ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:10.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.293+0000 7fb698f1c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb694072590 0x7fb694072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:10.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.293+0000 7fb698f1c640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb694072100 con 0x7fb69406e350 2023-12-04T05:48:10.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.293+0000 7fb698f1c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6940715b0 con 0x7fb694072590 2023-12-04T05:48:10.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.293+0000 7fb698f1c640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb69406f190 con 0x7fb69406e730 2023-12-04T05:48:10.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.293+0000 7fb692d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb694072590 0x7fb694072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:10.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.293+0000 7fb692d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb694072590 0x7fb694072a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57670/0 (socket says 172.21.15.107:57670) 2023-12-04T05:48:10.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.293+0000 7fb692d9d640 1 -- 172.21.15.107:0/1825677042 learned_addr learned my addr 172.21.15.107:0/1825677042 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:48:10.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.293+0000 7fb691d9b640 1 --2- 172.21.15.107:0/1825677042 >> v2:172.21.15.195:3300/0 conn(0x7fb69406e730 0x7fb694070ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:10.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.293+0000 7fb692d9d640 1 -- 172.21.15.107:0/1825677042 >> v2:172.21.15.195:3300/0 conn(0x7fb69406e730 msgr2=0x7fb694070ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:10.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.293+0000 7fb692d9d640 1 --2- 172.21.15.107:0/1825677042 >> v2:172.21.15.195:3300/0 conn(0x7fb69406e730 0x7fb694070ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.293+0000 7fb692d9d640 1 -- 172.21.15.107:0/1825677042 >> v1:172.21.15.195:6789/0 conn(0x7fb69406e350 legacy=0x7fb6941576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T05:48:10.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.293+0000 7fb692d9d640 1 -- 172.21.15.107:0/1825677042 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69406f470 con 0x7fb694072590 2023-12-04T05:48:10.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.293+0000 7fb691d9b640 1 --2- 172.21.15.107:0/1825677042 >> v2:172.21.15.195:3300/0 conn(0x7fb69406e730 0x7fb694070ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T05:48:10.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb692d9d640 1 --2- 172.21.15.107:0/1825677042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb694072590 0x7fb694072a00 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fb6940c8570 tx=0x7fb68c01f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e2cd08ba7ac4d13 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:10.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb69159a640 1 -- 172.21.15.107:0/1825677042 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c0af400 con 0x7fb694072590 2023-12-04T05:48:10.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb69159a640 1 -- 172.21.15.107:0/1825677042 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68c0ab050 con 0x7fb694072590 2023-12-04T05:48:10.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb69159a640 1 -- 172.21.15.107:0/1825677042 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c0af780 con 0x7fb694072590 2023-12-04T05:48:10.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb698f1c640 1 -- 172.21.15.107:0/1825677042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb694072590 msgr2=0x7fb694072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:10.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb698f1c640 1 --2- 172.21.15.107:0/1825677042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb694072590 0x7fb694072a00 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fb6940c8570 tx=0x7fb68c01f180 comp rx=0 tx=0).stop 2023-12-04T05:48:10.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb698f1c640 1 -- 172.21.15.107:0/1825677042 shutdown_connections 2023-12-04T05:48:10.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb698f1c640 1 --2- 172.21.15.107:0/1825677042 >> v2:172.21.15.195:3300/0 conn(0x7fb69406e730 0x7fb694070ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb698f1c640 1 --2- 172.21.15.107:0/1825677042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb694072590 0x7fb694072a00 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb698f1c640 1 -- 172.21.15.107:0/1825677042 >> 172.21.15.107:0/1825677042 conn(0x7fb6940cc440 msgr2=0x7fb6940cc850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:10.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb698f1c640 1 -- 172.21.15.107:0/1825677042 shutdown_connections 2023-12-04T05:48:10.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb698f1c640 1 -- 172.21.15.107:0/1825677042 wait complete. 2023-12-04T05:48:10.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb698f1c640 1 Processor -- start 2023-12-04T05:48:10.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb698f1c640 1 -- start start 2023-12-04T05:48:10.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb698f1c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb69406e350 0x7fb6941438f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:10.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb698f1c640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb69406e730 0x7fb69414c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:10.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb698f1c640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb694072590 0x7fb6941423b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:10.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb698f1c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6941443f0 con 0x7fb69406e350 2023-12-04T05:48:10.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb698f1c640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6941f0010 con 0x7fb69406e730 2023-12-04T05:48:10.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb698f1c640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6941f04a0 con 0x7fb694072590 2023-12-04T05:48:10.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb69259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb69406e350 0x7fb6941438f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:10.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb69259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb69406e350 0x7fb6941438f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57684/0 (socket says 172.21.15.107:57684) 2023-12-04T05:48:10.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb69259c640 1 -- 172.21.15.107:0/499991466 learned_addr learned my addr 172.21.15.107:0/499991466 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-12-04T05:48:10.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb691d9b640 1 --2- 172.21.15.107:0/499991466 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb69406e730 0x7fb69414c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:10.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb69259c640 1 -- 172.21.15.107:0/499991466 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb694072590 msgr2=0x7fb6941423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:10.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb69259c640 1 --2- 172.21.15.107:0/499991466 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb694072590 0x7fb6941423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb69259c640 1 -- 172.21.15.107:0/499991466 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb69406e730 msgr2=0x7fb69414c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:10.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb69259c640 1 --2- 172.21.15.107:0/499991466 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb69406e730 0x7fb69414c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb69259c640 1 -- 172.21.15.107:0/499991466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c009b30 con 0x7fb69406e350 2023-12-04T05:48:10.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.297+0000 7fb69259c640 1 --2- 172.21.15.107:0/499991466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb69406e350 0x7fb6941438f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb68400b410 tx=0x7fb68400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:10.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.301+0000 7fb682ffd640 1 -- 172.21.15.107:0/499991466 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb684004020 con 0x7fb69406e350 2023-12-04T05:48:10.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.301+0000 7fb682ffd640 1 -- 172.21.15.107:0/499991466 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb684004180 con 0x7fb69406e350 2023-12-04T05:48:10.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.301+0000 7fb698f1c640 1 -- 172.21.15.107:0/499991466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb694144110 con 0x7fb69406e350 2023-12-04T05:48:10.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.301+0000 7fb682ffd640 1 -- 172.21.15.107:0/499991466 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb684011630 con 0x7fb69406e350 2023-12-04T05:48:10.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.301+0000 7fb698f1c640 1 -- 172.21.15.107:0/499991466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6941ef9c0 con 0x7fb69406e350 2023-12-04T05:48:10.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.301+0000 7fb698f1c640 1 -- 172.21.15.107:0/499991466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb694067ec0 con 0x7fb69406e350 2023-12-04T05:48:10.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.301+0000 7fb682ffd640 1 -- 172.21.15.107:0/499991466 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 86170+0+0 (secure 0 0 0) 0x7fb684011790 con 0x7fb69406e350 2023-12-04T05:48:10.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.301+0000 7fb682ffd640 1 --2- 172.21.15.107:0/499991466 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb6740668d0 0x7fb674068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:10.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.301+0000 7fb682ffd640 1 -- 172.21.15.107:0/499991466 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fb684084400 con 0x7fb69406e350 2023-12-04T05:48:10.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.301+0000 7fb691d9b640 1 --2- 172.21.15.107:0/499991466 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb6740668d0 0x7fb674068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T05:48:10.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.301+0000 7fb691d9b640 1 --2- 172.21.15.107:0/499991466 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb6740668d0 0x7fb674068d90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb688006fd0 tx=0x7fb688008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T05:48:10.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.305+0000 7fb682ffd640 1 -- 172.21.15.107:0/499991466 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb6840513e0 con 0x7fb69406e350 2023-12-04T05:48:10.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.461+0000 7fb698f1c640 1 -- 172.21.15.107:0/499991466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb6940c88d0 con 0x7fb69406e350 2023-12-04T05:48:10.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.461+0000 7fb682ffd640 1 -- 172.21.15.107:0/499991466 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v27) v1 ==== 80+0+494 (secure 0 0 0) 0x7fb684015200 con 0x7fb69406e350 2023-12-04T05:48:10.465 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6839/3319682203 2023-12-05T05:45:54.798833+0000 2023-12-04T05:48:10.465 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6838/3319682203 2023-12-05T05:45:54.798833+0000 2023-12-04T05:48:10.465 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.195:6836/2222928136 2023-12-05T05:45:09.028534+0000 2023-12-04T05:48:10.465 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.195:6833/569093002 2023-12-05T05:43:14.736286+0000 2023-12-04T05:48:10.465 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.195:6832/569093002 2023-12-05T05:43:14.736286+0000 2023-12-04T05:48:10.466 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.195:6834/2222928136 2023-12-05T05:45:09.028534+0000 2023-12-04T05:48:10.466 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.195:6836/3769196693 2023-12-05T05:42:14.714882+0000 2023-12-04T05:48:10.466 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.195:6834/3769196693 2023-12-05T05:42:14.714882+0000 2023-12-04T05:48:10.467 INFO:teuthology.orchestra.run.smithi107.stderr:listed 8 entries 2023-12-04T05:48:10.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.461+0000 7fb698f1c640 1 -- 172.21.15.107:0/499991466 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb6740668d0 msgr2=0x7fb674068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:10.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.465+0000 7fb698f1c640 1 --2- 172.21.15.107:0/499991466 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb6740668d0 0x7fb674068d90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb688006fd0 tx=0x7fb688008040 comp rx=0 tx=0).stop 2023-12-04T05:48:10.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.465+0000 7fb698f1c640 1 -- 172.21.15.107:0/499991466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb69406e350 msgr2=0x7fb6941438f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T05:48:10.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.465+0000 7fb698f1c640 1 --2- 172.21.15.107:0/499991466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb69406e350 0x7fb6941438f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb68400b410 tx=0x7fb68400b8d0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.465+0000 7fb698f1c640 1 -- 172.21.15.107:0/499991466 shutdown_connections 2023-12-04T05:48:10.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.465+0000 7fb698f1c640 1 --2- 172.21.15.107:0/499991466 >> [v2:172.21.15.107:6832/2693048026,v1:172.21.15.107:6833/2693048026] conn(0x7fb6740668d0 0x7fb674068d90 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.465+0000 7fb698f1c640 1 --2- 172.21.15.107:0/499991466 >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fb694072590 0x7fb6941423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.465+0000 7fb698f1c640 1 --2- 172.21.15.107:0/499991466 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb69406e730 0x7fb69414c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.465+0000 7fb698f1c640 1 --2- 172.21.15.107:0/499991466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb69406e350 0x7fb6941438f0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:10.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.465+0000 7fb698f1c640 1 -- 172.21.15.107:0/499991466 >> 172.21.15.107:0/499991466 conn(0x7fb6940cc440 msgr2=0x7fb6940c6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T05:48:10.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.465+0000 7fb698f1c640 1 -- 172.21.15.107:0/499991466 shutdown_connections 2023-12-04T05:48:10.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:10.465+0000 7fb698f1c640 1 -- 172.21.15.107:0/499991466 wait complete. 2023-12-04T05:48:10.482 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi107.front.sepia.ceph.com... 2023-12-04T05:48:10.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:48:10.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-04T05:48:10.504 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T05:48:10.504 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2023-12-04T05:48:10.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:48:10.505 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-04T05:48:12.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T05:48:12.162 INFO:tasks.ceph.osd.0:Sent signal 15 2023-12-04T05:48:12.162 INFO:tasks.ceph.osd.1:Sent signal 15 2023-12-04T05:48:12.162 INFO:tasks.ceph.osd.2:Sent signal 15 2023-12-04T05:48:12.163 INFO:tasks.ceph.osd.3:Sent signal 15 2023-12-04T05:48:12.163 INFO:tasks.ceph.osd.4:Sent signal 15 2023-12-04T05:48:12.163 INFO:tasks.ceph.osd.5:Sent signal 15 2023-12-04T05:48:12.164 INFO:tasks.ceph.osd.6:Sent signal 15 2023-12-04T05:48:12.164 INFO:tasks.ceph.osd.7:Sent signal 15 2023-12-04T05:48:12.164 INFO:tasks.ceph.mds.c:Sent signal 15 2023-12-04T05:48:12.165 INFO:tasks.ceph.mds.e:Sent signal 15 2023-12-04T05:48:12.165 INFO:tasks.ceph.mds.f:Sent signal 15 2023-12-04T05:48:12.165 INFO:tasks.ceph.mon.a:Sent signal 15 2023-12-04T05:48:12.165 INFO:tasks.ceph.mon.b:Sent signal 15 2023-12-04T05:48:12.166 INFO:tasks.ceph.mon.c:Sent signal 15 2023-12-04T05:48:12.166 INFO:tasks.ceph.mgr.y:Sent signal 15 2023-12-04T05:48:12.166 INFO:tasks.ceph.mgr.x:Sent signal 15 2023-12-04T05:48:12.168 INFO:tasks.ceph.osd.4.smithi195.stderr:2023-12-04T05:48:12.164+0000 7f20f2fde640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38586) UID: 0 2023-12-04T05:48:12.168 INFO:tasks.ceph.osd.4.smithi195.stderr:2023-12-04T05:48:12.164+0000 7f20f2fde640 -1 osd.4 27 *** Got signal Terminated *** 2023-12-04T05:48:12.168 INFO:tasks.ceph.osd.4.smithi195.stderr:2023-12-04T05:48:12.164+0000 7f20f2fde640 -1 osd.4 27 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-04T05:48:12.168 INFO:tasks.ceph.osd.0.smithi107.stderr:2023-12-04T05:48:12.161+0000 7fde5856a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 38436) UID: 0 2023-12-04T05:48:12.168 INFO:tasks.ceph.osd.0.smithi107.stderr:2023-12-04T05:48:12.161+0000 7fde5856a640 -1 osd.0 27 *** Got signal Terminated *** 2023-12-04T05:48:12.169 INFO:tasks.ceph.osd.0.smithi107.stderr:2023-12-04T05:48:12.161+0000 7fde5856a640 -1 osd.0 27 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-04T05:48:12.169 INFO:tasks.ceph.osd.5.smithi195.stderr:2023-12-04T05:48:12.164+0000 7f2fc0442640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38589) UID: 0 2023-12-04T05:48:12.169 INFO:tasks.ceph.osd.5.smithi195.stderr:2023-12-04T05:48:12.164+0000 7f2fc0442640 -1 osd.5 27 *** Got signal Terminated *** 2023-12-04T05:48:12.169 INFO:tasks.ceph.osd.5.smithi195.stderr:2023-12-04T05:48:12.164+0000 7f2fc0442640 -1 osd.5 27 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-04T05:48:12.170 INFO:tasks.ceph.osd.1.smithi107.stderr:2023-12-04T05:48:12.161+0000 7f4954486640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 38438) UID: 0 2023-12-04T05:48:12.170 INFO:tasks.ceph.osd.1.smithi107.stderr:2023-12-04T05:48:12.161+0000 7f4954486640 -1 osd.1 27 *** Got signal Terminated *** 2023-12-04T05:48:12.170 INFO:tasks.ceph.osd.1.smithi107.stderr:2023-12-04T05:48:12.161+0000 7f4954486640 -1 osd.1 27 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-04T05:48:12.170 INFO:tasks.ceph.osd.2.smithi107.stderr:2023-12-04T05:48:12.161+0000 7fad69444640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 38440) UID: 0 2023-12-04T05:48:12.171 INFO:tasks.ceph.osd.2.smithi107.stderr:2023-12-04T05:48:12.161+0000 7fad69444640 -1 osd.2 27 *** Got signal Terminated *** 2023-12-04T05:48:12.171 INFO:tasks.ceph.osd.2.smithi107.stderr:2023-12-04T05:48:12.161+0000 7fad69444640 -1 osd.2 27 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-04T05:48:12.171 INFO:tasks.ceph.osd.6.smithi195.stderr:2023-12-04T05:48:12.164+0000 7fd570f0b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38593) UID: 0 2023-12-04T05:48:12.171 INFO:tasks.ceph.osd.6.smithi195.stderr:2023-12-04T05:48:12.164+0000 7fd570f0b640 -1 osd.6 27 *** Got signal Terminated *** 2023-12-04T05:48:12.171 INFO:tasks.ceph.osd.6.smithi195.stderr:2023-12-04T05:48:12.164+0000 7fd570f0b640 -1 osd.6 27 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-04T05:48:12.172 INFO:tasks.ceph.osd.3.smithi107.stderr:2023-12-04T05:48:12.161+0000 7f7758f2f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 38442) UID: 0 2023-12-04T05:48:12.172 INFO:tasks.ceph.osd.3.smithi107.stderr:2023-12-04T05:48:12.161+0000 7f7758f2f640 -1 osd.3 27 *** Got signal Terminated *** 2023-12-04T05:48:12.172 INFO:tasks.ceph.osd.3.smithi107.stderr:2023-12-04T05:48:12.161+0000 7f7758f2f640 -1 osd.3 27 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-04T05:48:12.172 INFO:tasks.ceph.osd.7.smithi195.stderr:2023-12-04T05:48:12.164+0000 7f82fa4a4640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38591) UID: 0 2023-12-04T05:48:12.173 INFO:tasks.ceph.osd.7.smithi195.stderr:2023-12-04T05:48:12.164+0000 7f82fa4a4640 -1 osd.7 27 *** Got signal Terminated *** 2023-12-04T05:48:12.173 INFO:tasks.ceph.osd.7.smithi195.stderr:2023-12-04T05:48:12.164+0000 7f82fa4a4640 -1 osd.7 27 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-04T05:48:12.173 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-12-04T05:48:12.161+0000 7f08026b8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mds -f --cluster ceph -i c (PID: 44013) UID: 0 2023-12-04T05:48:12.173 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-12-04T05:48:12.161+0000 7f08026b8640 -1 mds.c *** got signal Terminated *** 2023-12-04T05:48:12.174 INFO:tasks.ceph.mds.f.smithi195.stderr:2023-12-04T05:48:12.164+0000 7f5bdedc0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mds -f --cluster ceph -i f (PID: 43659) UID: 0 2023-12-04T05:48:12.174 INFO:tasks.ceph.mds.f.smithi195.stderr:2023-12-04T05:48:12.164+0000 7f5bdedc0640 -1 mds.f *** got signal Terminated *** 2023-12-04T05:48:12.174 INFO:tasks.ceph.mds.e.smithi107.stderr:2023-12-04T05:48:12.161+0000 7f95f335f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mds -f --cluster ceph -i e (PID: 44015) UID: 0 2023-12-04T05:48:12.174 INFO:tasks.ceph.mds.e.smithi107.stderr:2023-12-04T05:48:12.161+0000 7f95f335f640 -1 mds.e *** got signal Terminated *** 2023-12-04T05:48:12.175 INFO:tasks.ceph.mon.a.smithi107.stderr:2023-12-04T05:48:12.165+0000 7f1ad1477640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 38324) UID: 0 2023-12-04T05:48:12.175 INFO:tasks.ceph.mon.a.smithi107.stderr:2023-12-04T05:48:12.165+0000 7f1ad1477640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-04T05:48:12.176 INFO:tasks.ceph.mon.b.smithi195.stderr:2023-12-04T05:48:12.164+0000 7f6ce440f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i b (PID: 38266) UID: 0 2023-12-04T05:48:12.176 INFO:tasks.ceph.mon.b.smithi195.stderr:2023-12-04T05:48:12.164+0000 7f6ce440f640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2023-12-04T05:48:12.176 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:48:12.165+0000 7fdb3bd89640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster ceph -i y (PID: 38329) UID: 0 2023-12-04T05:48:12.176 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-12-04T05:48:12.165+0000 7fdb3bd89640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-12-04T05:48:12.176 INFO:tasks.ceph.mon.c.smithi195.stderr:2023-12-04T05:48:12.164+0000 7ff007cee640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i c (PID: 38268) UID: 0 2023-12-04T05:48:12.177 INFO:tasks.ceph.mon.c.smithi195.stderr:2023-12-04T05:48:12.164+0000 7ff007cee640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-12-04T05:48:12.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:12.225+0000 7fd55e55a640 1 Processor -- start 2023-12-04T05:48:12.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:12.225+0000 7fd55e55a640 1 -- start start 2023-12-04T05:48:12.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:12.225+0000 7fd55e55a640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd558154020 0x7fd55815a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:12.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:12.225+0000 7fd55e55a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5580d01c0 0x7fd558071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:12.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:12.225+0000 7fd55e55a640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815ad50 con 0x7fd558072220 2023-12-04T05:48:12.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:12.225+0000 7fd55e55a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5580ca230 con 0x7fd5580d01c0 2023-12-04T05:48:12.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:12.225+0000 7fd55e55a640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5580c6770 con 0x7fd558154020 2023-12-04T05:48:12.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:12.225+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5580d01c0 msgr2=0x7fd558071ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:12.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:12.225+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5580d01c0 0x7fd558071ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:12.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:12.225+0000 7fd55caf7640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd558072220 legacy=0x7fd5580c4c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:12.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:12.225+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd558154020 msgr2=0x7fd55815a810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:12.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:12.225+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd558154020 0x7fd55815a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:12.368 INFO:tasks.ceph.mgr.x.smithi195.stderr:daemon-helper: command crashed with signal 15 2023-12-04T05:48:12.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:12.429+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5580d01c0 msgr2=0x7fd558071ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:12.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:12.429+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5580d01c0 0x7fd558071ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:12.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:12.429+0000 7fd55caf7640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd558072220 legacy=0x7fd5580c4c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:12.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:12.429+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd558154020 msgr2=0x7fd55815a810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:12.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:12.429+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd558154020 0x7fd55815a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:12.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:12.833+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5580d01c0 msgr2=0x7fd558071ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:12.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:12.833+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5580d01c0 0x7fd558071ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:12.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:12.833+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd558154020 msgr2=0x7fd55815a810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:12.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:12.833+0000 7fd55caf7640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd558072220 legacy=0x7fd5580c4c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:12.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:12.833+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd558154020 0x7fd55815a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:13.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:13.637+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5580d01c0 msgr2=0x7fd558071ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:13.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:13.637+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5580d01c0 0x7fd558071ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:13.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:13.637+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd558154020 msgr2=0x7fd55815a810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:13.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:13.637+0000 7fd55caf7640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd558072220 legacy=0x7fd5580c4c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:13.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:13.637+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd558154020 0x7fd55815a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:15.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:15.225+0000 7fd555ffb640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd558154020 msgr2=0x7fd55815a810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:15.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:15.225+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd558154020 0x7fd55815a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:15.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:15.225+0000 7fd555ffb640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5580d01c0 msgr2=0x7fd558071ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:15.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:15.225+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5580d01c0 0x7fd558071ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:15.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:15.225+0000 7fd555ffb640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd558072220 legacy=0x7fd5580c4c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:15.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:15.225+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd5440036d0 0x7fd544005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:15.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:15.225+0000 7fd555ffb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815d710 con 0x7fd544006230 2023-12-04T05:48:15.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:15.225+0000 7fd555ffb640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd5440036d0 2023-12-04T05:48:15.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:15.225+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544006230 legacy=0x7fd5440086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:15.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:15.225+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd5440036d0 msgr2=0x7fd544005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:15.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:15.225+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd5440036d0 0x7fd544005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:15.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:15.429+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544006230 legacy=0x7fd5440086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:15.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:15.429+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd5440036d0 msgr2=0x7fd544005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:15.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:15.429+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd5440036d0 0x7fd544005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:15.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:15.833+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544006230 legacy=0x7fd5440086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:15.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:15.833+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd5440036d0 msgr2=0x7fd544005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:15.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:15.833+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd5440036d0 0x7fd544005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:16.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:16.637+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544006230 legacy=0x7fd5440086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:16.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:16.637+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd5440036d0 msgr2=0x7fd544005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:16.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:16.637+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd5440036d0 0x7fd544005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:18.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.225+0000 7fd555ffb640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd5440036d0 msgr2=0x7fd544005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:18.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.225+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd5440036d0 0x7fd544005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:18.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.225+0000 7fd555ffb640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544006230 legacy=0x7fd5440086f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:18.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.225+0000 7fd55caf7640 1 -- reap_dead start 2023-12-04T05:48:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.225+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd558154020 0x7fd54400a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.225+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.225+0000 7fd555ffb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815ad50 con 0x7fd558072220 2023-12-04T05:48:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.225+0000 7fd555ffb640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd544003b20 2023-12-04T05:48:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.225+0000 7fd555ffb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54400abe0 con 0x7fd558154020 2023-12-04T05:48:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.225+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd558154020 msgr2=0x7fd54400a6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.225+0000 7fd55caf7640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.225+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd558072220 legacy=0x7fd544009f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.225+0000 7fd55caf7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:18.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.225+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd558154020 0x7fd54400a6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:18.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.429+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd558154020 msgr2=0x7fd54400a6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:18.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.429+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd558072220 legacy=0x7fd544009f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:18.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.429+0000 7fd55caf7640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:18.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.429+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd558154020 0x7fd54400a6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:18.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.429+0000 7fd55caf7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:18.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.833+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd558154020 msgr2=0x7fd54400a6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:18.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.833+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd558072220 legacy=0x7fd544009f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:18.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.833+0000 7fd55caf7640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:18.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.833+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd558154020 0x7fd54400a6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:18.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:18.833+0000 7fd55caf7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:19.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:19.637+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd558154020 msgr2=0x7fd54400a6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:19.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:19.637+0000 7fd55caf7640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:19.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:19.637+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd558072220 legacy=0x7fd544009f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:19.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:19.637+0000 7fd55caf7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:19.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:19.637+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd558154020 0x7fd54400a6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:21.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:21.229+0000 7fd555ffb640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd558154020 msgr2=0x7fd54400a6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:21.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:21.229+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd558154020 0x7fd54400a6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:21.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:21.229+0000 7fd555ffb640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:21.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:21.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:21.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:21.229+0000 7fd555ffb640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd558072220 legacy=0x7fd544009f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:21.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:21.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544009800 0x7fd544013280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:21.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:21.229+0000 7fd555ffb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815d710 con 0x7fd544009360 2023-12-04T05:48:21.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:21.229+0000 7fd555ffb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd544009800 2023-12-04T05:48:21.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:21.229+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544009360 legacy=0x7fd5440086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:21.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:21.229+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544009800 msgr2=0x7fd544013280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:21.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:21.229+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544009800 0x7fd544013280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:21.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:21.433+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544009800 msgr2=0x7fd544013280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:21.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:21.433+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544009360 legacy=0x7fd5440086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:21.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:21.433+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544009800 0x7fd544013280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:21.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:21.837+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544009800 msgr2=0x7fd544013280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:21.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:21.837+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544009360 legacy=0x7fd5440086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:21.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:21.837+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544009800 0x7fd544013280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:22.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:22.641+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544009800 msgr2=0x7fd544013280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:22.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:22.641+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544009360 legacy=0x7fd5440086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:22.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:22.641+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544009800 0x7fd544013280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:24.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.229+0000 7fd555ffb640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544009800 msgr2=0x7fd544013280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:24.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544009800 0x7fd544013280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:24.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.229+0000 7fd555ffb640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544009360 legacy=0x7fd5440086f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:24.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440137c0 0x7fd544013ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:24.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:24.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.229+0000 7fd55caf7640 1 -- reap_dead start 2023-12-04T05:48:24.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.229+0000 7fd555ffb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815ad50 con 0x7fd544013350 2023-12-04T05:48:24.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.229+0000 7fd555ffb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd5440137c0 2023-12-04T05:48:24.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.229+0000 7fd555ffb640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400abe0 con 0x7fd544003b20 2023-12-04T05:48:24.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.229+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440137c0 msgr2=0x7fd544013ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:24.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.229+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440137c0 0x7fd544013ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:24.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.229+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:24.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.229+0000 7fd55caf7640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544013350 legacy=0x7fd5440086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:24.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.229+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:24.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.433+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440137c0 msgr2=0x7fd544013ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:24.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.433+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440137c0 0x7fd544013ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:24.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.433+0000 7fd55caf7640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544013350 legacy=0x7fd5440086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:24.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.433+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:24.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.433+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:24.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.837+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440137c0 msgr2=0x7fd544013ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:24.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.837+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440137c0 0x7fd544013ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:24.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.837+0000 7fd55caf7640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544013350 legacy=0x7fd5440086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:24.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.837+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:24.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:24.837+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:25.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:25.641+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440137c0 msgr2=0x7fd544013ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:25.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:25.641+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440137c0 0x7fd544013ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:25.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:25.641+0000 7fd55caf7640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544013350 legacy=0x7fd5440086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:25.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:25.641+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:25.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:25.641+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:27.229+0000 7fd555ffb640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:27.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:27.229+0000 7fd555ffb640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440137c0 msgr2=0x7fd544013ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:27.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440137c0 0x7fd544013ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:27.229+0000 7fd555ffb640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544013350 legacy=0x7fd5440086f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:27.229+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 0x7fd54400a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:27.229+0000 7fd555ffb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815d710 con 0x7fd54400e250 2023-12-04T05:48:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:27.229+0000 7fd555ffb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd54400a030 2023-12-04T05:48:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:27.229+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400e250 legacy=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:27.229+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 msgr2=0x7fd54400a410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:27.229+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 0x7fd54400a410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:27.433+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400e250 legacy=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:27.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:27.433+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 msgr2=0x7fd54400a410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:27.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:27.433+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 0x7fd54400a410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:27.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:27.837+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400e250 legacy=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:27.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:27.837+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 msgr2=0x7fd54400a410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:27.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:27.837+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 0x7fd54400a410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:28.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:28.641+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400e250 legacy=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:28.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:28.641+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 msgr2=0x7fd54400a410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:28.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:28.641+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 0x7fd54400a410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:30.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.229+0000 7fd555ffb640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 msgr2=0x7fd54400a410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:30.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.229+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 0x7fd54400a410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:30.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.229+0000 7fd555ffb640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400e250 legacy=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:30.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.229+0000 7fd55caf7640 1 -- reap_dead start 2023-12-04T05:48:30.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440137c0 0x7fd544013ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:30.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.229+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 0x7fd54400a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:30.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:30.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.229+0000 7fd555ffb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd5440137c0 2023-12-04T05:48:30.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.229+0000 7fd555ffb640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400abe0 con 0x7fd544003b20 2023-12-04T05:48:30.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.229+0000 7fd555ffb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd544012ea0 con 0x7fd54400a030 2023-12-04T05:48:30.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.229+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440137c0 msgr2=0x7fd544013ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:30.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.229+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440137c0 0x7fd544013ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:30.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.229+0000 7fd55caf7640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:30.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.229+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 msgr2=0x7fd54400a410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:30.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.229+0000 7fd55caf7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:30.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.229+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 0x7fd54400a410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:30.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.433+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440137c0 msgr2=0x7fd544013ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:30.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.433+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440137c0 0x7fd544013ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:30.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.433+0000 7fd55caf7640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:30.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.433+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 msgr2=0x7fd54400a410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:30.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.433+0000 7fd55caf7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:30.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.433+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 0x7fd54400a410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:30.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.837+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440137c0 msgr2=0x7fd544013ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:30.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.837+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440137c0 0x7fd544013ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:30.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.837+0000 7fd55caf7640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:30.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.837+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 msgr2=0x7fd54400a410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:30.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.837+0000 7fd55caf7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:30.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:30.837+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 0x7fd54400a410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:31.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:48:31.548 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:48:31.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:31.641+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440137c0 msgr2=0x7fd544013ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:31.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:31.641+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 msgr2=0x7fd54400a410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:31.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:31.641+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440137c0 0x7fd544013ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:31.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:31.641+0000 7fd55caf7640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:31.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:31.641+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 0x7fd54400a410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:31.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:31.641+0000 7fd55caf7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:33.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:33.225+0000 7fd555ffb640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 msgr2=0x7fd54400a410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:33.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:33.225+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 0x7fd54400a410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:33.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:33.225+0000 7fd555ffb640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:33.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:33.225+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:33.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:33.225+0000 7fd555ffb640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440137c0 msgr2=0x7fd544013ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:33.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:33.225+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440137c0 0x7fd544013ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:33.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:33.225+0000 7fd555ffb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815ad50 con 0x7fd5440066e0 2023-12-04T05:48:33.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:33.225+0000 7fd555ffb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815d710 con 0x7fd54400e250 2023-12-04T05:48:33.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:33.225+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd5440066e0 legacy=0x7fd5440140e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:33.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:33.225+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400e250 legacy=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:33.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:33.429+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd5440066e0 legacy=0x7fd5440140e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:33.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:33.429+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400e250 legacy=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:33.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:33.833+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd5440066e0 legacy=0x7fd5440140e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:33.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:33.833+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400e250 legacy=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:34.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:34.637+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd5440066e0 legacy=0x7fd5440140e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:34.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:34.637+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400e250 legacy=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:36.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:36.225+0000 7fd555ffb640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400e250 legacy=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:36.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:36.225+0000 7fd555ffb640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd5440066e0 legacy=0x7fd5440140e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:36.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:36.225+0000 7fd55caf7640 1 -- reap_dead start 2023-12-04T05:48:36.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:36.225+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:36.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:36.225+0000 7fd555ffb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd544008b30 con 0x7fd5440066e0 2023-12-04T05:48:36.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:36.225+0000 7fd555ffb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815ad50 con 0x7fd544015800 2023-12-04T05:48:36.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:36.225+0000 7fd555ffb640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd544003b20 2023-12-04T05:48:36.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:36.225+0000 7fd55caf7640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd5440066e0 legacy=0x7fd54400b230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:36.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:36.229+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:36.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:36.229+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544015800 legacy=0x7fd544017bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:36.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:36.229+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:36.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:36.429+0000 7fd55caf7640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd5440066e0 legacy=0x7fd54400b230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:36.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:36.433+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544015800 legacy=0x7fd544017bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:36.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:36.433+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:36.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:36.433+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:36.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:36.833+0000 7fd55caf7640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd5440066e0 legacy=0x7fd54400b230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:36.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:36.837+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544015800 legacy=0x7fd544017bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:36.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:36.837+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:36.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:36.837+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:37.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:37.637+0000 7fd55caf7640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd5440066e0 legacy=0x7fd54400b230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:37.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:37.641+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:37.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:37.641+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544015800 legacy=0x7fd544017bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:37.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:37.641+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:39.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:39.229+0000 7fd555ffb640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:39.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:39.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:39.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:39.229+0000 7fd555ffb640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544015800 legacy=0x7fd544017bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:39.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:39.229+0000 7fd555ffb640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd5440066e0 legacy=0x7fd54400b230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:39.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:39.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440150f0 0x7fd5440137c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:39.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:39.229+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013d00 0x7fd5440106d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:39.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:39.229+0000 7fd555ffb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd5440150f0 2023-12-04T05:48:39.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:39.229+0000 7fd555ffb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54400abe0 con 0x7fd544013d00 2023-12-04T05:48:39.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:39.229+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440150f0 msgr2=0x7fd5440137c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:39.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:39.229+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440150f0 0x7fd5440137c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:39.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:39.229+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013d00 msgr2=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:39.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:39.229+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013d00 0x7fd5440106d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:39.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:39.433+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440150f0 msgr2=0x7fd5440137c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:39.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:39.433+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440150f0 0x7fd5440137c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:39.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:39.433+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013d00 msgr2=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:39.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:39.433+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013d00 0x7fd5440106d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:39.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:39.837+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440150f0 msgr2=0x7fd5440137c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:39.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:39.837+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440150f0 0x7fd5440137c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:39.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:39.837+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013d00 msgr2=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:39.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:39.837+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013d00 0x7fd5440106d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:40.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:40.641+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440150f0 msgr2=0x7fd5440137c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:40.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:40.641+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440150f0 0x7fd5440137c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:40.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:40.641+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013d00 msgr2=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:40.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:40.641+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013d00 0x7fd5440106d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:42.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.229+0000 7fd555ffb640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013d00 msgr2=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:42.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.229+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013d00 0x7fd5440106d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:42.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.229+0000 7fd555ffb640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440150f0 msgr2=0x7fd5440137c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:42.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440150f0 0x7fd5440137c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:42.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:42.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.229+0000 7fd55caf7640 1 -- reap_dead start 2023-12-04T05:48:42.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.229+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013d00 0x7fd5440106d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:42.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.229+0000 7fd555ffb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815d710 con 0x7fd544015800 2023-12-04T05:48:42.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.229+0000 7fd555ffb640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd544003b20 2023-12-04T05:48:42.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.229+0000 7fd555ffb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54400abe0 con 0x7fd544013d00 2023-12-04T05:48:42.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.229+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:42.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.229+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544015800 legacy=0x7fd54400d950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:42.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.229+0000 7fd55caf7640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013d00 msgr2=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:42.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.229+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:42.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.229+0000 7fd55caf7640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013d00 0x7fd5440106d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:42.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.433+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:42.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.433+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544015800 legacy=0x7fd54400d950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:42.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.433+0000 7fd55caf7640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013d00 msgr2=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:42.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.433+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:42.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.433+0000 7fd55caf7640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013d00 0x7fd5440106d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:42.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.837+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544015800 legacy=0x7fd54400d950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:42.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.837+0000 7fd55caf7640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013d00 msgr2=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:42.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.837+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:42.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.837+0000 7fd55caf7640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013d00 0x7fd5440106d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:42.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:42.837+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:43.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:43.641+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:43.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:43.641+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544015800 legacy=0x7fd54400d950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:43.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:43.641+0000 7fd55caf7640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013d00 msgr2=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:43.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:43.641+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:43.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:43.641+0000 7fd55caf7640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013d00 0x7fd5440106d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:45.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:45.229+0000 7fd555ffb640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013d00 msgr2=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:45.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:45.229+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013d00 0x7fd5440106d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:45.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:45.229+0000 7fd555ffb640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:45.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:45.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:45.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:45.229+0000 7fd555ffb640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544015800 legacy=0x7fd54400d950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:45.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:45.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440066e0 0x7fd5440137c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:45.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:45.229+0000 7fd555ffb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd544008b30 con 0x7fd54400b1f0 2023-12-04T05:48:45.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:45.229+0000 7fd555ffb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd5440066e0 2023-12-04T05:48:45.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:45.229+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440066e0 msgr2=0x7fd5440137c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:45.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:45.229+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400b1f0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:45.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:45.229+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440066e0 0x7fd5440137c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:45.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:45.433+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400b1f0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:45.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:45.433+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440066e0 msgr2=0x7fd5440137c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:45.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:45.433+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440066e0 0x7fd5440137c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:45.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:45.837+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400b1f0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:45.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:45.837+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440066e0 msgr2=0x7fd5440137c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:45.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:45.837+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440066e0 0x7fd5440137c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:46.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:46.641+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400b1f0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:46.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:46.641+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440066e0 msgr2=0x7fd5440137c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:46.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:46.641+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440066e0 0x7fd5440137c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:48.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.229+0000 7fd555ffb640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440066e0 msgr2=0x7fd5440137c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:48.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440066e0 0x7fd5440137c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:48.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.229+0000 7fd555ffb640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400b1f0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:48.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440150f0 0x7fd544018c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:48.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.229+0000 7fd55caf7640 1 -- reap_dead start 2023-12-04T05:48:48.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:48.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.229+0000 7fd555ffb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815d710 con 0x7fd544015800 2023-12-04T05:48:48.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.229+0000 7fd555ffb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd5440150f0 2023-12-04T05:48:48.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.229+0000 7fd555ffb640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400abe0 con 0x7fd544003b20 2023-12-04T05:48:48.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.229+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440150f0 msgr2=0x7fd544018c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:48.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.229+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440150f0 0x7fd544018c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:48.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.229+0000 7fd55caf7640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544015800 legacy=0x7fd544015be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:48.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.229+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:48.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.229+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:48.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.433+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440150f0 msgr2=0x7fd544018c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:48.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.433+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440150f0 0x7fd544018c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:48.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.433+0000 7fd55caf7640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544015800 legacy=0x7fd544015be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:48.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.433+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:48.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.433+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:48.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.837+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440150f0 msgr2=0x7fd544018c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:48.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.837+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440150f0 0x7fd544018c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:48.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.837+0000 7fd55caf7640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544015800 legacy=0x7fd544015be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:48.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.837+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:48.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:48.837+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:49.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:49.641+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440150f0 msgr2=0x7fd544018c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:49.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:49.641+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440150f0 0x7fd544018c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:49.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:49.641+0000 7fd55caf7640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544015800 legacy=0x7fd544015be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:49.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:49.641+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:49.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:49.641+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:51.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:51.229+0000 7fd555ffb640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:51.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:51.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:51.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:51.229+0000 7fd555ffb640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440150f0 msgr2=0x7fd544018c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:51.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:51.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440150f0 0x7fd544018c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:51.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:51.229+0000 7fd555ffb640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544015800 legacy=0x7fd544015be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:51.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:51.229+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 0x7fd5440106d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:51.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:51.229+0000 7fd555ffb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd544008b30 con 0x7fd54400b1f0 2023-12-04T05:48:51.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:51.229+0000 7fd555ffb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd5440147d0 2023-12-04T05:48:51.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:51.229+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400b1f0 legacy=0x7fd5440162f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:51.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:51.229+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 msgr2=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:51.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:51.229+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 0x7fd5440106d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:51.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:51.433+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400b1f0 legacy=0x7fd5440162f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:51.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:51.433+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 msgr2=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:51.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:51.433+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 0x7fd5440106d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:51.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:51.837+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400b1f0 legacy=0x7fd5440162f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:51.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:51.837+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 msgr2=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:51.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:51.837+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 0x7fd5440106d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:52.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:52.641+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400b1f0 legacy=0x7fd5440162f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:52.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:52.641+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 msgr2=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:52.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:52.641+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 0x7fd5440106d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:54.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:54.229+0000 7fd555ffb640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 msgr2=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:54.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:54.229+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 0x7fd5440106d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:54.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:54.229+0000 7fd555ffb640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400b1f0 legacy=0x7fd5440162f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:54.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:54.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:54.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:54.229+0000 7fd55caf7640 1 -- reap_dead start 2023-12-04T05:48:54.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:54.229+0000 7fd555ffb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815d710 con 0x7fd544014bb0 2023-12-04T05:48:54.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:54.229+0000 7fd555ffb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd544008b30 con 0x7fd544015800 2023-12-04T05:48:54.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:54.229+0000 7fd555ffb640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd544003b20 2023-12-04T05:48:54.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:54.229+0000 7fd55caf7640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544014bb0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:54.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:54.229+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544015800 legacy=0x7fd544015be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:54.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:54.229+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:54.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:54.229+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:54.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:54.433+0000 7fd55caf7640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544014bb0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:54.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:54.433+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544015800 legacy=0x7fd544015be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:54.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:54.433+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:54.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:54.433+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:54.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:54.837+0000 7fd55caf7640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544014bb0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:54.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:54.837+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:54.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:54.837+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544015800 legacy=0x7fd544015be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:54.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:54.837+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:55.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:55.641+0000 7fd55caf7640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544014bb0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:48:55.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:55.641+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:48:55.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:55.641+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544015800 legacy=0x7fd544015be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:48:55.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:55.641+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:57.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:57.229+0000 7fd555ffb640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:57.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:57.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd544008e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:48:57.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:57.229+0000 7fd555ffb640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544015800 legacy=0x7fd544015be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:57.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:57.229+0000 7fd555ffb640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544014bb0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:48:57.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:57.229+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 0x7fd5440106d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:57.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:57.229+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544014f90 0x7fd54400c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:48:57.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:57.229+0000 7fd555ffb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd544014f90 2023-12-04T05:48:57.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:57.229+0000 7fd555ffb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54400abe0 con 0x7fd5440147d0 2023-12-04T05:48:57.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:57.229+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544014f90 msgr2=0x7fd54400c5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:57.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:57.229+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544014f90 0x7fd54400c5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:57.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:57.229+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 msgr2=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:57.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:57.229+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 0x7fd5440106d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:48:57.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:57.433+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544014f90 msgr2=0x7fd54400c5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:57.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:57.433+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544014f90 0x7fd54400c5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:57.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:57.433+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 msgr2=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:57.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:57.433+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 0x7fd5440106d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:48:57.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:57.837+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544014f90 msgr2=0x7fd54400c5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:57.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:57.837+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544014f90 0x7fd54400c5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:57.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:57.837+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 msgr2=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:57.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:57.837+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 0x7fd5440106d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:48:58.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:58.641+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544014f90 msgr2=0x7fd54400c5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:48:58.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:58.641+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544014f90 0x7fd54400c5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:48:58.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:58.641+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 msgr2=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:48:58.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:48:58.641+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 0x7fd5440106d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:00.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.233+0000 7fd555ffb640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 msgr2=0x7fd5440106d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:00.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.233+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 0x7fd5440106d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:00.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.233+0000 7fd555ffb640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544014f90 msgr2=0x7fd54400c5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:00.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.233+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544014f90 0x7fd54400c5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:00.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.233+0000 7fd55caf7640 1 -- reap_dead start 2023-12-04T05:49:00.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.233+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 0x7fd544008db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:00.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.233+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 0x7fd54400e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:00.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.233+0000 7fd555ffb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815ad50 con 0x7fd544014bb0 2023-12-04T05:49:00.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.233+0000 7fd555ffb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd544003b20 2023-12-04T05:49:00.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.233+0000 7fd555ffb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54400abe0 con 0x7fd5440147d0 2023-12-04T05:49:00.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.233+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544014bb0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:00.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.233+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:00.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.233+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 0x7fd544008db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:00.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.233+0000 7fd55caf7640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:00.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.233+0000 7fd55caf7640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:00.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.437+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:00.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.437+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544014bb0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:00.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.437+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 0x7fd544008db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:00.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.437+0000 7fd55caf7640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:00.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.437+0000 7fd55caf7640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:00.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.841+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:00.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.841+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544014bb0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:00.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.841+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 0x7fd544008db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:00.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.841+0000 7fd55caf7640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:00.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:00.841+0000 7fd55caf7640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:01.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:01.645+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:01.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:01.645+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544014bb0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:01.645+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 0x7fd544008db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:01.645+0000 7fd55caf7640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:01.645+0000 7fd55caf7640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:01.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:49:01.679 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:49:03.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:03.233+0000 7fd555ffb640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:03.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:03.233+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440147d0 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:03.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:03.233+0000 7fd555ffb640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 msgr2=0x7fd544008db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:03.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:03.233+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 0x7fd544008db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:03.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:03.233+0000 7fd555ffb640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544014bb0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:03.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:03.233+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 0x7fd54400fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:03.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:03.233+0000 7fd555ffb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815d710 con 0x7fd54400ec00 2023-12-04T05:49:03.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:03.233+0000 7fd555ffb640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd54400dcf0 2023-12-04T05:49:03.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:03.233+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 msgr2=0x7fd54400fff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:03.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:03.233+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400ec00 legacy=0x7fd54400d5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:03.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:03.233+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 0x7fd54400fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:03.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:03.437+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400ec00 legacy=0x7fd54400d5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:03.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:03.437+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 msgr2=0x7fd54400fff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:03.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:03.437+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 0x7fd54400fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:03.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:03.841+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 msgr2=0x7fd54400fff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:03.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:03.841+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400ec00 legacy=0x7fd54400d5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:03.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:03.841+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 0x7fd54400fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:04.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:04.645+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 msgr2=0x7fd54400fff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:04.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:04.645+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400ec00 legacy=0x7fd54400d5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:04.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:04.645+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 0x7fd54400fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:06.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.233+0000 7fd555ffb640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 msgr2=0x7fd54400fff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:06.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.233+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 0x7fd54400fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:06.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.233+0000 7fd555ffb640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400ec00 legacy=0x7fd54400d5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:06.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.233+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 0x7fd54400fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:06.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.233+0000 7fd55caf7640 1 -- reap_dead start 2023-12-04T05:49:06.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.233+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 0x7fd54400e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:06.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.233+0000 7fd555ffb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815ad50 con 0x7fd544010530 2023-12-04T05:49:06.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.233+0000 7fd555ffb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd544003b20 2023-12-04T05:49:06.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.233+0000 7fd555ffb640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400abe0 con 0x7fd54400dcf0 2023-12-04T05:49:06.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.233+0000 7fd55caf7640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:06.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.233+0000 7fd55caf7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:06.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.233+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544010530 legacy=0x7fd544018fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:06.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.233+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 msgr2=0x7fd54400fff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:06.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.233+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 0x7fd54400fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:06.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.437+0000 7fd55caf7640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:06.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.437+0000 7fd55caf7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:06.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.437+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 msgr2=0x7fd54400fff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:06.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.437+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544010530 legacy=0x7fd544018fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:06.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.437+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 0x7fd54400fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:06.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.841+0000 7fd55caf7640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:06.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.841+0000 7fd55caf7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:06.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.841+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 msgr2=0x7fd54400fff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:06.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.841+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544010530 legacy=0x7fd544018fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:06.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:06.841+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 0x7fd54400fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:07.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:07.645+0000 7fd55caf7640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:07.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:07.645+0000 7fd55caf7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:07.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:07.645+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 msgr2=0x7fd54400fff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:07.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:07.645+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544010530 legacy=0x7fd544018fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:07.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:07.645+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 0x7fd54400fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:09.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:09.233+0000 7fd555ffb640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 msgr2=0x7fd54400fff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:09.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:09.233+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 0x7fd54400fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:09.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:09.233+0000 7fd555ffb640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:09.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:09.233+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:09.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:09.233+0000 7fd555ffb640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544010530 legacy=0x7fd544018fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:09.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:09.233+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544008d80 0x7fd54400a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:09.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:09.233+0000 7fd555ffb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815d710 con 0x7fd54400efe0 2023-12-04T05:49:09.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:09.233+0000 7fd555ffb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd544008d80 2023-12-04T05:49:09.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:09.233+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400efe0 legacy=0x7fd54400f3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:09.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:09.233+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544008d80 msgr2=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:09.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:09.233+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544008d80 0x7fd54400a030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:09.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:09.437+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400efe0 legacy=0x7fd54400f3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:09.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:09.437+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544008d80 msgr2=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:09.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:09.437+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544008d80 0x7fd54400a030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:09.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:09.841+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400efe0 legacy=0x7fd54400f3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:09.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:09.841+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544008d80 msgr2=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:09.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:09.841+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544008d80 0x7fd54400a030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:10.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:10.645+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400efe0 legacy=0x7fd54400f3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:10.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:10.645+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544008d80 msgr2=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:10.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:10.645+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544008d80 0x7fd54400a030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:12.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.233+0000 7fd555ffb640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544008d80 msgr2=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:12.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.233+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544008d80 0x7fd54400a030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:12.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.233+0000 7fd555ffb640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400efe0 legacy=0x7fd54400f3c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:12.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.233+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 0x7fd54400e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:12.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.233+0000 7fd55caf7640 1 -- reap_dead start 2023-12-04T05:49:12.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.233+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 0x7fd54400d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:12.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.233+0000 7fd555ffb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815ad50 con 0x7fd544010530 2023-12-04T05:49:12.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.233+0000 7fd555ffb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd544003b20 2023-12-04T05:49:12.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.233+0000 7fd555ffb640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400abe0 con 0x7fd54400dcf0 2023-12-04T05:49:12.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.233+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:12.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.233+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:12.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.233+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 msgr2=0x7fd54400d5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:12.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.233+0000 7fd55caf7640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544010530 legacy=0x7fd54400fad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:12.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.233+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 0x7fd54400d5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:12.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.437+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:12.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.437+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:12.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.437+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 msgr2=0x7fd54400d5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:12.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.437+0000 7fd55caf7640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544010530 legacy=0x7fd54400fad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:12.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.437+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 0x7fd54400d5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:12.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.840+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:12.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.840+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:12.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.840+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 msgr2=0x7fd54400d5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:12.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.840+0000 7fd55caf7640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544010530 legacy=0x7fd54400fad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:12.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:12.840+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 0x7fd54400d5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:13.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:13.644+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:13.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:13.644+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:13.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:13.644+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 msgr2=0x7fd54400d5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:13.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:13.644+0000 7fd55caf7640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544010530 legacy=0x7fd54400fad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:13.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:13.644+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 0x7fd54400d5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:15.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:15.232+0000 7fd555ffb640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 msgr2=0x7fd54400d5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:15.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:15.232+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400dcf0 0x7fd54400d5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:15.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:15.232+0000 7fd555ffb640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:15.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:15.232+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544003b20 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:15.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:15.232+0000 7fd555ffb640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544010530 legacy=0x7fd54400fad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:15.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:15.232+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544008d80 0x7fd54400a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:15.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:15.232+0000 7fd555ffb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815d710 con 0x7fd54400efe0 2023-12-04T05:49:15.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:15.232+0000 7fd555ffb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd544008d80 2023-12-04T05:49:15.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:15.232+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400efe0 legacy=0x7fd54400f3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:15.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:15.232+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544008d80 msgr2=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:15.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:15.232+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544008d80 0x7fd54400a030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:15.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:15.436+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400efe0 legacy=0x7fd54400f3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:15.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:15.436+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544008d80 msgr2=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:15.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:15.436+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544008d80 0x7fd54400a030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:15.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:15.840+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400efe0 legacy=0x7fd54400f3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:15.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:15.840+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544008d80 msgr2=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:15.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:15.840+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544008d80 0x7fd54400a030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:16.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:16.644+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400efe0 legacy=0x7fd54400f3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:16.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:16.644+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544008d80 msgr2=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:16.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:16.644+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544008d80 0x7fd54400a030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:18.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.232+0000 7fd555ffb640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544008d80 msgr2=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:18.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.232+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544008d80 0x7fd54400a030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:18.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.232+0000 7fd555ffb640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400efe0 legacy=0x7fd54400f3c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:18.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.232+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440177f0 0x7fd544019be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:18.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.232+0000 7fd55caf7640 1 -- reap_dead start 2023-12-04T05:49:18.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.232+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400e6c0 0x7fd54400dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:18.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.232+0000 7fd555ffb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815ad50 con 0x7fd544003b20 2023-12-04T05:49:18.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.232+0000 7fd555ffb640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd54400e6c0 2023-12-04T05:49:18.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.232+0000 7fd555ffb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54400abe0 con 0x7fd5440177f0 2023-12-04T05:49:18.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.232+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544003b20 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:18.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.232+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440177f0 msgr2=0x7fd544019be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:18.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.232+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440177f0 0x7fd544019be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:18.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.232+0000 7fd55caf7640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400e6c0 msgr2=0x7fd54400dbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:18.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.232+0000 7fd55caf7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400e6c0 0x7fd54400dbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:18.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.436+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544003b20 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:18.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.436+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440177f0 msgr2=0x7fd544019be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:18.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.436+0000 7fd55caf7640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400e6c0 msgr2=0x7fd54400dbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:18.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.436+0000 7fd55caf7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400e6c0 0x7fd54400dbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:18.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.436+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440177f0 0x7fd544019be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:18.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.840+0000 7fd55caf7640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400e6c0 msgr2=0x7fd54400dbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:18.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.840+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544003b20 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:18.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.840+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440177f0 msgr2=0x7fd544019be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:18.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.840+0000 7fd55caf7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400e6c0 0x7fd54400dbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:18.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:18.840+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440177f0 0x7fd544019be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:19.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:19.644+0000 7fd55caf7640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400e6c0 msgr2=0x7fd54400dbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:19.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:19.644+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544003b20 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:19.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:19.644+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440177f0 msgr2=0x7fd544019be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:19.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:19.644+0000 7fd55caf7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400e6c0 0x7fd54400dbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:19.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:19.644+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440177f0 0x7fd544019be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:21.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:21.236+0000 7fd555ffb640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440177f0 msgr2=0x7fd544019be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:21.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:21.236+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440177f0 0x7fd544019be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:21.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:21.236+0000 7fd555ffb640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400e6c0 msgr2=0x7fd54400dbc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:21.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:21.236+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400e6c0 0x7fd54400dbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:21.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:21.236+0000 7fd555ffb640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544003b20 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:21.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:21.236+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400fff0 0x7fd544010460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:21.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:21.236+0000 7fd555ffb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815d710 con 0x7fd54400b5d0 2023-12-04T05:49:21.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:21.236+0000 7fd555ffb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd54400fff0 2023-12-04T05:49:21.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:21.236+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd54400fff0 msgr2=0x7fd544010460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:21.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:21.236+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400b5d0 legacy=0x7fd5440137c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:21.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:21.236+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400fff0 0x7fd544010460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:21.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:21.440+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd54400fff0 msgr2=0x7fd544010460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:21.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:21.440+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400b5d0 legacy=0x7fd5440137c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:21.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:21.440+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400fff0 0x7fd544010460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:21.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:21.844+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400b5d0 legacy=0x7fd5440137c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:21.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:21.844+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd54400fff0 msgr2=0x7fd544010460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:21.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:21.844+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400fff0 0x7fd544010460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:22.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:22.648+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400b5d0 legacy=0x7fd5440137c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:22.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:22.648+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd54400fff0 msgr2=0x7fd544010460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:22.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:22.648+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400fff0 0x7fd544010460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:24.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.236+0000 7fd555ffb640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd54400fff0 msgr2=0x7fd544010460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:24.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.236+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400fff0 0x7fd544010460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:24.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.236+0000 7fd555ffb640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400b5d0 legacy=0x7fd5440137c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:24.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.236+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544014ee0 0x7fd54401a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:24.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.236+0000 7fd55caf7640 1 -- reap_dead start 2023-12-04T05:49:24.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.236+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440177f0 0x7fd544010460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:24.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.236+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd54400e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:24.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.236+0000 7fd555ffb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd544014ee0 2023-12-04T05:49:24.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.236+0000 7fd555ffb640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400abe0 con 0x7fd544003b20 2023-12-04T05:49:24.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.236+0000 7fd555ffb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd544012ea0 con 0x7fd5440177f0 2023-12-04T05:49:24.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.236+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544014ee0 msgr2=0x7fd54401a120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:24.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.236+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544014ee0 0x7fd54401a120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:24.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.236+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440177f0 msgr2=0x7fd544010460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:24.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.236+0000 7fd55caf7640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:24.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.236+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440177f0 0x7fd544010460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:24.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.236+0000 7fd55caf7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:24.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.440+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544014ee0 msgr2=0x7fd54401a120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:24.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.440+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544014ee0 0x7fd54401a120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:24.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.440+0000 7fd55caf7640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:24.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.440+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440177f0 msgr2=0x7fd544010460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:24.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.440+0000 7fd55caf7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:24.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.440+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440177f0 0x7fd544010460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:24.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.844+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544014ee0 msgr2=0x7fd54401a120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:24.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.844+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544014ee0 0x7fd54401a120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:24.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.844+0000 7fd55caf7640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:24.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.844+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440177f0 msgr2=0x7fd544010460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.844+0000 7fd55caf7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:24.844+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440177f0 0x7fd544010460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:25.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:25.648+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544014ee0 msgr2=0x7fd54401a120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:25.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:25.648+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544014ee0 0x7fd54401a120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:25.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:25.648+0000 7fd55caf7640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:25.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:25.648+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440177f0 msgr2=0x7fd544010460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:25.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:25.648+0000 7fd55caf7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:25.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:25.648+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440177f0 0x7fd544010460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:27.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:27.236+0000 7fd555ffb640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440177f0 msgr2=0x7fd544010460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:27.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:27.236+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440177f0 0x7fd544010460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:27.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:27.236+0000 7fd555ffb640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:27.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:27.236+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544003b20 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:27.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:27.236+0000 7fd555ffb640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544014ee0 msgr2=0x7fd54401a120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:27.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:27.236+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544014ee0 0x7fd54401a120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:27.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:27.236+0000 7fd555ffb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815ad50 con 0x7fd5440172d0 2023-12-04T05:49:27.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:27.236+0000 7fd555ffb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815d710 con 0x7fd54400b5d0 2023-12-04T05:49:27.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:27.236+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd5440172d0 legacy=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:27.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:27.236+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400b5d0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:27.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:27.440+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd5440172d0 legacy=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:27.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:27.440+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400b5d0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:27.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:27.844+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd5440172d0 legacy=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:27.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:27.844+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400b5d0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:28.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:28.648+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd5440172d0 legacy=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:28.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:28.648+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400b5d0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:30.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:30.236+0000 7fd555ffb640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400b5d0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:30.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:30.236+0000 7fd555ffb640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd5440172d0 legacy=0x7fd544013780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:30.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:30.236+0000 7fd55caf7640 1 -- reap_dead start 2023-12-04T05:49:30.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:30.236+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 0x7fd54400e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:30.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:30.236+0000 7fd555ffb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd544008b30 con 0x7fd54400dbc0 2023-12-04T05:49:30.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:30.236+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400dbc0 legacy=0x7fd54401a660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:30.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:30.236+0000 7fd555ffb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815ad50 con 0x7fd54400b5d0 2023-12-04T05:49:30.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:30.236+0000 7fd555ffb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd5440177f0 2023-12-04T05:49:30.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:30.236+0000 7fd55caf7640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:30.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:30.236+0000 7fd55caf7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:30.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:30.236+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400b5d0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:30.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:30.440+0000 7fd55caf7640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:30.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:30.440+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400dbc0 legacy=0x7fd54401a660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:30.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:30.440+0000 7fd55caf7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:30.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:30.440+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400b5d0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:30.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:30.844+0000 7fd55caf7640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:30.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:30.844+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400dbc0 legacy=0x7fd54401a660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:30.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:30.844+0000 7fd55caf7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:30.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:30.844+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400b5d0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:31.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:31.648+0000 7fd55caf7640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:31.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:31.648+0000 7fd55caf7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:31.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:31.648+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400dbc0 legacy=0x7fd54401a660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:31.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:31.648+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400b5d0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:31.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:49:31.810 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:49:33.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:33.236+0000 7fd555ffb640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:33.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:33.236+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:33.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:33.236+0000 7fd555ffb640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400b5d0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:33.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:33.236+0000 7fd555ffb640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400dbc0 legacy=0x7fd54401a660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:33.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:33.236+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544010460 0x7fd544019be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:33.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:33.236+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54401a120 0x7fd544013780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:33.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:33.236+0000 7fd555ffb640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd544010460 2023-12-04T05:49:33.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:33.236+0000 7fd555ffb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54400abe0 con 0x7fd54401a120 2023-12-04T05:49:33.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:33.236+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54401a120 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:33.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:33.236+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544010460 msgr2=0x7fd544019be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:33.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:33.236+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54401a120 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:33.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:33.236+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544010460 0x7fd544019be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:33.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:33.440+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54401a120 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:33.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:33.440+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544010460 msgr2=0x7fd544019be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:33.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:33.440+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54401a120 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:33.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:33.440+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544010460 0x7fd544019be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:33.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:33.844+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544010460 msgr2=0x7fd544019be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:33.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:33.844+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54401a120 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:33.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:33.844+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544010460 0x7fd544019be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:33.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:33.844+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54401a120 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:34.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:34.648+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544010460 msgr2=0x7fd544019be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:34.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:34.648+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54401a120 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:34.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:34.648+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544010460 0x7fd544019be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:34.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:34.648+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54401a120 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:36.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.236+0000 7fd555ffb640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54401a120 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:36.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.236+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54401a120 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.236+0000 7fd555ffb640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544010460 msgr2=0x7fd544019be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.236+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544010460 0x7fd544019be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.236+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544005b30 0x7fd544013cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.236+0000 7fd55caf7640 1 -- reap_dead start 2023-12-04T05:49:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.236+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 0x7fd544013780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.236+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544016ef0 0x7fd54400e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.236+0000 7fd555ffb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd5440177f0 2023-12-04T05:49:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.236+0000 7fd555ffb640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400abe0 con 0x7fd544016ef0 2023-12-04T05:49:36.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.236+0000 7fd555ffb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd544012ea0 con 0x7fd544005b30 2023-12-04T05:49:36.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.236+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:36.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.236+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:36.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.236+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544005b30 msgr2=0x7fd544013cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:36.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.236+0000 7fd55caf7640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544016ef0 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:36.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.236+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544005b30 0x7fd544013cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:36.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.236+0000 7fd55caf7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544016ef0 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:36.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.440+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:36.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.440+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:36.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.440+0000 7fd55caf7640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544016ef0 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:36.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.440+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544005b30 msgr2=0x7fd544013cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:36.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.440+0000 7fd55caf7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544016ef0 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:36.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.440+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544005b30 0x7fd544013cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:36.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.844+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:36.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.844+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:36.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.844+0000 7fd55caf7640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544016ef0 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:36.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.844+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544005b30 msgr2=0x7fd544013cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:36.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.844+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544005b30 0x7fd544013cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:36.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:36.844+0000 7fd55caf7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544016ef0 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:37.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:37.648+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:37.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:37.648+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:37.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:37.648+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544005b30 msgr2=0x7fd544013cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:37.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:37.648+0000 7fd55caf7640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544016ef0 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:37.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:37.648+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544005b30 0x7fd544013cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:37.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:37.648+0000 7fd55caf7640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544016ef0 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:39.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:39.236+0000 7fd555ffb640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544005b30 msgr2=0x7fd544013cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:39.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:39.236+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544005b30 0x7fd544013cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:39.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:39.236+0000 7fd555ffb640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544016ef0 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:39.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:39.236+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544016ef0 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:39.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:39.236+0000 7fd555ffb640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:39.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:39.236+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:39.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:39.236+0000 7fd555ffb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815d710 con 0x7fd54400da60 2023-12-04T05:49:39.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:39.236+0000 7fd555ffb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd544008b30 con 0x7fd544010460 2023-12-04T05:49:39.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:39.236+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400da60 legacy=0x7fd544014200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:39.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:39.236+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544010460 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:39.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:39.440+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400da60 legacy=0x7fd544014200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:39.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:39.440+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544010460 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:39.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:39.844+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400da60 legacy=0x7fd544014200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:39.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:39.844+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544010460 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:40.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:40.648+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400da60 legacy=0x7fd544014200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:40.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:40.648+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544010460 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:42.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:42.236+0000 7fd555ffb640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544010460 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:42.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:42.236+0000 7fd555ffb640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400da60 legacy=0x7fd544014200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:42.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:42.236+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 0x7fd544013780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:42.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:42.236+0000 7fd55caf7640 1 -- reap_dead start 2023-12-04T05:49:42.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:42.236+0000 7fd555ffb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815ad50 con 0x7fd544005b30 2023-12-04T05:49:42.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:42.236+0000 7fd555ffb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815d710 con 0x7fd544010460 2023-12-04T05:49:42.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:42.236+0000 7fd555ffb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd5440177f0 2023-12-04T05:49:42.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:42.236+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:42.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:42.236+0000 7fd55caf7640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544005b30 legacy=0x7fd544013cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:42.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:42.236+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:42.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:42.236+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544010460 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:42.440+0000 7fd55caf7640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544005b30 legacy=0x7fd544013cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:42.440+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:42.440+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:42.440+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544010460 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:42.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:42.844+0000 7fd55caf7640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544005b30 legacy=0x7fd544013cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:42.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:42.844+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:42.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:42.844+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:42.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:42.844+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544010460 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:43.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:43.648+0000 7fd55caf7640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544005b30 legacy=0x7fd544013cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:43.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:43.648+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:43.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:43.648+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:43.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:43.648+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544010460 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:45.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:45.232+0000 7fd555ffb640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:45.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:45.232+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5440177f0 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:45.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:45.232+0000 7fd555ffb640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544010460 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:45.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:45.232+0000 7fd555ffb640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544005b30 legacy=0x7fd544013cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:45.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:45.232+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400da60 0x7fd54400e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:45.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:45.232+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400d5e0 0x7fd544019be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:45.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:45.236+0000 7fd555ffb640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd54400da60 2023-12-04T05:49:45.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:45.236+0000 7fd555ffb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54400abe0 con 0x7fd54400d5e0 2023-12-04T05:49:45.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:45.236+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400d5e0 msgr2=0x7fd544019be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:45.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:45.236+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400da60 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:45.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:45.236+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400d5e0 0x7fd544019be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:45.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:45.236+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400da60 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:45.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:45.440+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400da60 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:45.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:45.440+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400d5e0 msgr2=0x7fd544019be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:45.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:45.440+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400da60 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:45.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:45.440+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400d5e0 0x7fd544019be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:45.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:45.844+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400da60 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:45.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:45.844+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400d5e0 msgr2=0x7fd544019be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:45.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:45.844+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400da60 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:45.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:45.844+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400d5e0 0x7fd544019be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:46.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:46.648+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400da60 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:46.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:46.648+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400d5e0 msgr2=0x7fd544019be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:46.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:46.648+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400da60 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:46.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:46.648+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400d5e0 0x7fd544019be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:48.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:48.236+0000 7fd555ffb640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400d5e0 msgr2=0x7fd544019be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:48.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:48.236+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400d5e0 0x7fd544019be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:48.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:48.236+0000 7fd555ffb640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400da60 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:48.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:48.236+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400da60 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:48.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:48.236+0000 7fd55caf7640 1 -- reap_dead start 2023-12-04T05:49:48.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:48.236+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440143d0 0x7fd5440147d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:48.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:48.236+0000 7fd555ffb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd544008b30 con 0x7fd544010460 2023-12-04T05:49:48.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:48.236+0000 7fd555ffb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815ad50 con 0x7fd54400da20 2023-12-04T05:49:48.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:48.236+0000 7fd555ffb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd5440143d0 2023-12-04T05:49:48.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:48.236+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544010460 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:48.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:48.236+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440143d0 msgr2=0x7fd5440147d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:48.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:48.236+0000 7fd55caf7640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400da20 legacy=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:48.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:48.236+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440143d0 0x7fd5440147d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:48.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:48.440+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544010460 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:48.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:48.440+0000 7fd55caf7640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400da20 legacy=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:48.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:48.440+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440143d0 msgr2=0x7fd5440147d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:48.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:48.440+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440143d0 0x7fd5440147d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:48.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:48.844+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544010460 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:48.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:48.844+0000 7fd55caf7640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400da20 legacy=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:48.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:48.844+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440143d0 msgr2=0x7fd5440147d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:48.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:48.844+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440143d0 0x7fd5440147d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:49.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:49.648+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544010460 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:49.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:49.648+0000 7fd55caf7640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400da20 legacy=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:49.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:49.648+0000 7fd557fff640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440143d0 msgr2=0x7fd5440147d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:49.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:49.648+0000 7fd557fff640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440143d0 0x7fd5440147d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:51.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:51.240+0000 7fd555ffb640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440143d0 msgr2=0x7fd5440147d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:51.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:51.240+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440143d0 0x7fd5440147d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:51.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:51.240+0000 7fd555ffb640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd54400da20 legacy=0x7fd544013780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:51.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:51.240+0000 7fd555ffb640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544010460 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:51.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:51.240+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544013e90 0x7fd54400e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:51.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:51.240+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544016ef0 0x7fd544011ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:51.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:51.240+0000 7fd555ffb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd544016ef0 2023-12-04T05:49:51.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:51.240+0000 7fd555ffb640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400abe0 con 0x7fd544013e90 2023-12-04T05:49:51.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:51.240+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544016ef0 msgr2=0x7fd544011ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:51.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:51.240+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544016ef0 0x7fd544011ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:51.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:51.240+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544013e90 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:51.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:51.240+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544013e90 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:51.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:51.444+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544016ef0 msgr2=0x7fd544011ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:51.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:51.444+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544016ef0 0x7fd544011ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:51.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:51.444+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544013e90 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:51.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:51.444+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544013e90 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:51.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:51.848+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544016ef0 msgr2=0x7fd544011ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:51.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:51.848+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544016ef0 0x7fd544011ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:51.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:51.848+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544013e90 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:51.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:51.848+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544013e90 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:52.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:52.652+0000 7fd5577fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544016ef0 msgr2=0x7fd544011ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:52.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:52.652+0000 7fd5577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544016ef0 0x7fd544011ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:52.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:52.652+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544013e90 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:52.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:52.652+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544013e90 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:54.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.240+0000 7fd555ffb640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544013e90 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:54.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.240+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544013e90 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:54.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.240+0000 7fd555ffb640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd544016ef0 msgr2=0x7fd544011ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:54.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.240+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd544016ef0 0x7fd544011ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:54.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.240+0000 7fd55caf7640 1 -- reap_dead start 2023-12-04T05:49:54.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.240+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544013e90 0x7fd54400e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:54.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.240+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 0x7fd544013780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:54.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.240+0000 7fd555ffb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815d710 con 0x7fd544010460 2023-12-04T05:49:54.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.240+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544010460 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:54.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.240+0000 7fd555ffb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd54400da20 2023-12-04T05:49:54.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.240+0000 7fd555ffb640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400abe0 con 0x7fd544013e90 2023-12-04T05:49:54.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.240+0000 7fd55caf7640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:54.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.240+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544013e90 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:54.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.240+0000 7fd55caf7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:54.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.240+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544013e90 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:54.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.444+0000 7fd55caf7640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:54.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.444+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544010460 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:54.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.444+0000 7fd55caf7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:54.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.444+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544013e90 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:54.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.444+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544013e90 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:54.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.848+0000 7fd55caf7640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:54.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.848+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544010460 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:54.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.848+0000 7fd55caf7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:54.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.848+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544013e90 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:54.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:54.848+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544013e90 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:55.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:55.652+0000 7fd55caf7640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:49:55.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:55.652+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544010460 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:49:55.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:55.652+0000 7fd55caf7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:55.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:55.652+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544013e90 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:49:55.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:55.652+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544013e90 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:49:57.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:57.240+0000 7fd555ffb640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd544013e90 msgr2=0x7fd54400e6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:57.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:57.240+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd544013e90 0x7fd54400e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:57.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:57.240+0000 7fd555ffb640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:57.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:57.240+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:49:57.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:57.240+0000 7fd555ffb640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd544010460 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:49:57.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:57.240+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440149e0 0x7fd544019800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:49:57.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:57.240+0000 7fd555ffb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd544008b30 con 0x7fd544016ef0 2023-12-04T05:49:57.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:57.240+0000 7fd555ffb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd5440149e0 2023-12-04T05:49:57.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:57.240+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440149e0 msgr2=0x7fd544019800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:57.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:57.240+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544016ef0 legacy=0x7fd5440142d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:57.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:57.240+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440149e0 0x7fd544019800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:49:57.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:57.444+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544016ef0 legacy=0x7fd5440142d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:57.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:57.444+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440149e0 msgr2=0x7fd544019800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:57.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:57.444+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440149e0 0x7fd544019800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:49:57.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:57.848+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440149e0 msgr2=0x7fd544019800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:57.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:57.848+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544016ef0 legacy=0x7fd5440142d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:57.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:57.848+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440149e0 0x7fd544019800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:49:58.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:58.652+0000 7fd557fff640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544016ef0 legacy=0x7fd5440142d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:49:58.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:58.652+0000 7fd5577fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440149e0 msgr2=0x7fd544019800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:49:58.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:49:58.652+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440149e0 0x7fd544019800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:00.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.240+0000 7fd555ffb640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440149e0 msgr2=0x7fd544019800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:00.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.240+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd5440149e0 0x7fd544019800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:00.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.240+0000 7fd555ffb640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544016ef0 legacy=0x7fd5440142d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:00.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.240+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 0x7fd544013780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:00.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.240+0000 7fd55caf7640 1 -- reap_dead start 2023-12-04T05:50:00.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.240+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013cc0 0x7fd544010420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:00.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.240+0000 7fd555ffb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815d710 con 0x7fd54400e6c0 2023-12-04T05:50:00.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.240+0000 7fd555ffb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd54400da20 2023-12-04T05:50:00.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.240+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:00.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.240+0000 7fd555ffb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54400abe0 con 0x7fd544013cc0 2023-12-04T05:50:00.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.240+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:00.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.240+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400e6c0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:00.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.240+0000 7fd55caf7640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013cc0 msgr2=0x7fd544010420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:00.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.240+0000 7fd55caf7640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013cc0 0x7fd544010420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:00.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.444+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:00.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.444+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400e6c0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:00.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.444+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:00.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.444+0000 7fd55caf7640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013cc0 msgr2=0x7fd544010420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:00.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.444+0000 7fd55caf7640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013cc0 0x7fd544010420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:00.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.848+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400e6c0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:00.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.848+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:00.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.848+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:00.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.848+0000 7fd55caf7640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013cc0 msgr2=0x7fd544010420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:00.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:00.848+0000 7fd55caf7640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013cc0 0x7fd544010420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:01.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:01.652+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:01.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:01.652+0000 7fd5577fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400e6c0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:01.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:01.652+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:01.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:01.652+0000 7fd55caf7640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013cc0 msgr2=0x7fd544010420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:01.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:01.652+0000 7fd55caf7640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013cc0 0x7fd544010420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:02.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:50:02.155 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:50:03.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:03.240+0000 7fd555ffb640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013cc0 msgr2=0x7fd544010420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:03.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:03.240+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd544013cc0 0x7fd544010420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:03.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:03.240+0000 7fd555ffb640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:03.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:03.240+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:03.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:03.240+0000 7fd555ffb640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd54400e6c0 legacy=0x7fd54400a030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:03.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:03.240+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd5440142d0 0x7fd5440146b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:03.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:03.240+0000 7fd555ffb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd544008b30 con 0x7fd544016ef0 2023-12-04T05:50:03.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:03.240+0000 7fd555ffb640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd5440142d0 2023-12-04T05:50:03.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:03.240+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd5440142d0 msgr2=0x7fd5440146b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:03.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:03.240+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544016ef0 legacy=0x7fd5440177c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:03.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:03.240+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd5440142d0 0x7fd5440146b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:03.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:03.444+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd5440142d0 msgr2=0x7fd5440146b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:03.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:03.444+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544016ef0 legacy=0x7fd5440177c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:03.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:03.444+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd5440142d0 0x7fd5440146b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:03.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:03.848+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544016ef0 legacy=0x7fd5440177c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:03.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:03.848+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd5440142d0 msgr2=0x7fd5440146b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:03.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:03.848+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd5440142d0 0x7fd5440146b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:04.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:04.652+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544016ef0 legacy=0x7fd5440177c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:04.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:04.652+0000 7fd557fff640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd5440142d0 msgr2=0x7fd5440146b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:04.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:04.652+0000 7fd557fff640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd5440142d0 0x7fd5440146b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:06.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.240+0000 7fd555ffb640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd5440142d0 msgr2=0x7fd5440146b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:06.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.240+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd5440142d0 0x7fd5440146b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:06.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.240+0000 7fd555ffb640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544016ef0 legacy=0x7fd5440177c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:06.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.240+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 0x7fd544013780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:06.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.240+0000 7fd55caf7640 1 -- reap_dead start 2023-12-04T05:50:06.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.240+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400e6c0 0x7fd544013cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:06.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.240+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 0x7fd544010420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:06.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.240+0000 7fd555ffb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400af60 con 0x7fd54400da20 2023-12-04T05:50:06.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.240+0000 7fd555ffb640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54400abe0 con 0x7fd54400e6c0 2023-12-04T05:50:06.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.240+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:06.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.240+0000 7fd555ffb640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd544012ea0 con 0x7fd54400a030 2023-12-04T05:50:06.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.240+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:06.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.240+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400e6c0 msgr2=0x7fd544013cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:06.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.240+0000 7fd55caf7640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 msgr2=0x7fd544010420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:06.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.240+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400e6c0 0x7fd544013cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:06.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.240+0000 7fd55caf7640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 0x7fd544010420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:06.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.444+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:06.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.444+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:06.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.444+0000 7fd55caf7640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 msgr2=0x7fd544010420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:06.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.444+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400e6c0 msgr2=0x7fd544013cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:06.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.444+0000 7fd55caf7640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 0x7fd544010420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:06.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.444+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400e6c0 0x7fd544013cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:06.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.848+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:06.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.848+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:06.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.848+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400e6c0 msgr2=0x7fd544013cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:06.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.848+0000 7fd55caf7640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 msgr2=0x7fd544010420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:06.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.848+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400e6c0 0x7fd544013cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:06.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:06.848+0000 7fd55caf7640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 0x7fd544010420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:07.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:07.652+0000 7fd557fff640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:07.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:07.652+0000 7fd557fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:07.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:07.652+0000 7fd55caf7640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 msgr2=0x7fd544010420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:07.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:07.652+0000 7fd5577fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400e6c0 msgr2=0x7fd544013cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:07.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:07.652+0000 7fd5577fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400e6c0 0x7fd544013cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:07.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:07.652+0000 7fd55caf7640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 0x7fd544010420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:09.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:09.240+0000 7fd555ffb640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 msgr2=0x7fd544010420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:09.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:09.240+0000 7fd555ffb640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7fd54400a030 0x7fd544010420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:09.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:09.240+0000 7fd555ffb640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7fd54400e6c0 msgr2=0x7fd544013cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:09.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:09.240+0000 7fd555ffb640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7fd54400e6c0 0x7fd544013cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:09.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:09.240+0000 7fd555ffb640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 msgr2=0x7fd544013780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:09.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:09.240+0000 7fd555ffb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd54400da20 0x7fd544013780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:09.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:09.240+0000 7fd555ffb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55815d710 con 0x7fd5440142d0 2023-12-04T05:50:09.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:09.240+0000 7fd555ffb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd544008b30 con 0x7fd544016ef0 2023-12-04T05:50:09.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:09.240+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd5440142d0 legacy=0x7fd5440146b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:09.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:09.240+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544016ef0 legacy=0x7fd5440177c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:09.444+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd5440142d0 legacy=0x7fd5440146b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:09.444+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544016ef0 legacy=0x7fd5440177c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:09.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:09.848+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd5440142d0 legacy=0x7fd5440146b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:09.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:09.848+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544016ef0 legacy=0x7fd5440177c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:10.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:10.652+0000 7fd557fff640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fd5440142d0 legacy=0x7fd5440146b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:10.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:10.652+0000 7fd5577fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fd544016ef0 legacy=0x7fd5440177c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:12.116 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-04T05:50:12.116 ERROR:tasks.mds_thrash.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/mds_thrash.py", line 124, in _run self.do_thrash() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/mds_thrash.py", line 257, in do_thrash self.wait_for_stable() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/mds_thrash.py", line 188, in wait_for_stable status = self.fs.status() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/filesystem.py", line 396, in status return FSStatus(self.mon_manager, epoch) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/filesystem.py", line 88, in __init__ self.map = json.loads(self.get_ceph_cmd_stdout(*cmd)) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json' 2023-12-04T05:50:12.127 INFO:tasks.mds_thrash:done joining 2023-12-04T05:50:12.128 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2023-12-04T05:50:13.621 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2023-12-04T05:50:13.641 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-04T05:50:13.641 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-04T05:50:13.655 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336380k,nr_inodes=4084095,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274664k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=22896 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274660k,nr_inodes=818665,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274664k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2023-12-04T05:50:13.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:50:13.656 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-12-04T05:50:13.704 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-04T05:50:13.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:50:13.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-04T05:50:13.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T05:50:13.760 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link delete ceph-brx 2023-12-04T05:50:13.944 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-04T05:50:13.953 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-04T05:50:13.954 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-04T05:50:14.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:14.072+0000 7f1e0bca3640 1 Processor -- start 2023-12-04T05:50:14.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:14.072+0000 7f1e0bca3640 1 -- start start 2023-12-04T05:50:14.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:14.072+0000 7f1e0bca3640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1e0415a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:14.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:14.072+0000 7f1e0bca3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1e040c4c10 0x7f1e040c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:14.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:14.072+0000 7f1e0bca3640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415ad50 con 0x7f1e040d01c0 2023-12-04T05:50:14.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:14.072+0000 7f1e0bca3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e040ca230 con 0x7f1e040c4c10 2023-12-04T05:50:14.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:14.072+0000 7f1e0bca3640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e040c6770 con 0x7f1e04154020 2023-12-04T05:50:14.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:14.072+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1e040c4c10 msgr2=0x7f1e040c5010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:14.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:14.072+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1e040c4c10 0x7f1e040c5010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:14.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:14.076+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1e0415a810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:14.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:14.076+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1e040d01c0 legacy=0x7f1e04071ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:14.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:14.076+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1e0415a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:14.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:14.280+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1e040c4c10 msgr2=0x7f1e040c5010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:14.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:14.280+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1e040c4c10 0x7f1e040c5010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:14.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:14.280+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1e0415a810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:14.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:14.280+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1e040d01c0 legacy=0x7f1e04071ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:14.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:14.280+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1e0415a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:14.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:14.684+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1e040c4c10 msgr2=0x7f1e040c5010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:14.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:14.684+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1e040c4c10 0x7f1e040c5010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:14.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:14.684+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1e0415a810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:14.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:14.684+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1e040d01c0 legacy=0x7f1e04071ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:14.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:14.684+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1e0415a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:15.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:15.488+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1e040c4c10 msgr2=0x7f1e040c5010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:15.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:15.488+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1e040c4c10 0x7f1e040c5010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:15.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:15.488+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1e0415a810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:15.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:15.488+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1e040d01c0 legacy=0x7f1e04071ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:15.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:15.488+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1e0415a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:17.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:17.076+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1e0415a810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:17.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:17.076+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1e0415a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:17.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:17.076+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1e040c4c10 msgr2=0x7f1e040c5010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:17.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:17.076+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1e040c4c10 0x7f1e040c5010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:17.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:17.076+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1e040d01c0 legacy=0x7f1e04071ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:17.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:17.076+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8007380 0x7f1de8009840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:17.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:17.076+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415d710 con 0x7f1de80036a0 2023-12-04T05:50:17.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:17.076+0000 7f1dfb7fe640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8007380 2023-12-04T05:50:17.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:17.076+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de80036a0 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:17.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:17.076+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8007380 msgr2=0x7f1de8009840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:17.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:17.076+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8007380 0x7f1de8009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:17.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:17.280+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de80036a0 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:17.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:17.280+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8007380 msgr2=0x7f1de8009840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:17.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:17.280+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8007380 0x7f1de8009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:17.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:17.684+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de80036a0 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:17.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:17.684+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8007380 msgr2=0x7f1de8009840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:17.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:17.684+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8007380 0x7f1de8009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:18.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:18.488+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de80036a0 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:18.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:18.488+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8007380 msgr2=0x7f1de8009840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:18.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:18.488+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8007380 0x7f1de8009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:20.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:20.076+0000 7f1dfb7fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8007380 msgr2=0x7f1de8009840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:20.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:20.076+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8007380 0x7f1de8009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:20.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:20.076+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de80036a0 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:20.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:20.076+0000 7f1e0a240640 1 -- reap_dead start 2023-12-04T05:50:20.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:20.076+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80102a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:20.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:20.076+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415ad50 con 0x7f1de800a660 2023-12-04T05:50:20.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:20.076+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415d710 con 0x7f1e040d01c0 2023-12-04T05:50:20.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:20.076+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1e04154020 2023-12-04T05:50:20.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:20.076+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de800a660 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:20.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:20.076+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1e040d01c0 legacy=0x7f1de8009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:20.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:20.076+0000 7f1e0a240640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1de80102a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:20.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:20.076+0000 7f1e0a240640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80102a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:20.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:20.280+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de800a660 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:20.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:20.280+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1e040d01c0 legacy=0x7f1de8009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:20.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:20.280+0000 7f1e0a240640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1de80102a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:20.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:20.280+0000 7f1e0a240640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80102a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:20.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:20.684+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de800a660 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:20.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:20.684+0000 7f1e0a240640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1de80102a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:20.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:20.684+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1e040d01c0 legacy=0x7f1de8009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:20.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:20.684+0000 7f1e0a240640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80102a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:21.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:21.488+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de800a660 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:21.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:21.488+0000 7f1e0a240640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1de80102a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:21.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:21.488+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1e040d01c0 legacy=0x7f1de8009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:21.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:21.488+0000 7f1e0a240640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80102a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:23.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:23.080+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1de80102a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:23.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:23.080+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80102a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:23.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:23.080+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1e040d01c0 legacy=0x7f1de8009f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:23.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:23.080+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de800a660 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:23.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:23.080+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8007830 0x7f1de80107e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:23.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:23.080+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:23.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:23.080+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8007830 2023-12-04T05:50:23.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:23.080+0000 7f1dfb7fe640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80136f0 con 0x7f1de8010d20 2023-12-04T05:50:23.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:23.080+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8007830 msgr2=0x7f1de80107e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:23.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:23.080+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8007830 0x7f1de80107e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:23.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:23.080+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 msgr2=0x7f1de8013110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:23.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:23.080+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:23.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:23.284+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8007830 msgr2=0x7f1de80107e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:23.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:23.284+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8007830 0x7f1de80107e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:23.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:23.284+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 msgr2=0x7f1de8013110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:23.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:23.284+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:23.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:23.688+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8007830 msgr2=0x7f1de80107e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:23.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:23.688+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8007830 0x7f1de80107e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:23.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:23.688+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 msgr2=0x7f1de8013110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:23.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:23.688+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:24.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:24.492+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8007830 msgr2=0x7f1de80107e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:24.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:24.492+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8007830 0x7f1de80107e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:24.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:24.492+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 msgr2=0x7f1de8013110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:24.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:24.492+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:26.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.080+0000 7f1dfb7fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 msgr2=0x7f1de8013110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:26.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.080+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:26.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.080+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8007830 msgr2=0x7f1de80107e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:26.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.080+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8007830 0x7f1de80107e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:26.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.080+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80102a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:26.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.080+0000 7f1e0a240640 1 -- reap_dead start 2023-12-04T05:50:26.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.080+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:26.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.080+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de800ac20 con 0x7f1de800a660 2023-12-04T05:50:26.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.080+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1e04154020 2023-12-04T05:50:26.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.080+0000 7f1dfb7fe640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80136f0 con 0x7f1de8010d20 2023-12-04T05:50:26.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.080+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de800a660 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:26.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.080+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1de80102a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:26.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.080+0000 7f1e0a240640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 msgr2=0x7f1de8013110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:26.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.080+0000 7f1e0a240640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:26.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.080+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80102a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:26.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.284+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de800a660 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:26.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.284+0000 7f1e0a240640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 msgr2=0x7f1de8013110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:26.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.284+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1de80102a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:26.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.284+0000 7f1e0a240640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:26.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.284+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80102a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.688+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de800a660 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.688+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1de80102a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.688+0000 7f1e0a240640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 msgr2=0x7f1de8013110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.688+0000 7f1e0a240640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:26.688+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80102a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:27.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:27.492+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de800a660 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:27.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:27.492+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1de80102a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:27.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:27.492+0000 7f1e0a240640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 msgr2=0x7f1de8013110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:27.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:27.492+0000 7f1e0a240640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:27.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:27.492+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80102a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:29.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:29.080+0000 7f1dfb7fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 msgr2=0x7f1de8013110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:29.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:29.080+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:29.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:29.080+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1de80102a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:29.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:29.080+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80102a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:29.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:29.080+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de800a660 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:29.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:29.080+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800a040 0x7f1de8014030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:29.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:29.080+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415ad50 con 0x7f1de80108b0 2023-12-04T05:50:29.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:29.080+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de800a040 2023-12-04T05:50:29.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:29.080+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800a040 msgr2=0x7f1de8014030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:29.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:29.080+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de80108b0 legacy=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:29.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:29.080+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800a040 0x7f1de8014030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:29.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:29.284+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800a040 msgr2=0x7f1de8014030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:29.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:29.284+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800a040 0x7f1de8014030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:29.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:29.284+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de80108b0 legacy=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:29.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:29.688+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800a040 msgr2=0x7f1de8014030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:29.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:29.688+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800a040 0x7f1de8014030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:29.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:29.688+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de80108b0 legacy=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:30.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:30.492+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800a040 msgr2=0x7f1de8014030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:30.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:30.492+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800a040 0x7f1de8014030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:30.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:30.492+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de80108b0 legacy=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:32.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.080+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800a040 msgr2=0x7f1de8014030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:32.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.080+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800a040 0x7f1de8014030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:32.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.080+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de80108b0 legacy=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:32.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.080+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80102a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:32.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.080+0000 7f1e0a240640 1 -- reap_dead start 2023-12-04T05:50:32.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.080+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:32.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.080+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de800ac20 con 0x7f1de800a660 2023-12-04T05:50:32.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.080+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1e04154020 2023-12-04T05:50:32.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.080+0000 7f1dfb7fe640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80136f0 con 0x7f1de8010d20 2023-12-04T05:50:32.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.080+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de800a660 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:32.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.080+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1de80102a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:32.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.080+0000 7f1e0a240640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 msgr2=0x7f1de8013110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:32.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.080+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80102a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:32.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.080+0000 7f1e0a240640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:32.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:50:32.264 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:50:32.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.284+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de800a660 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:32.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.284+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1de80102a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:32.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.284+0000 7f1e0a240640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 msgr2=0x7f1de8013110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:32.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.284+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80102a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:32.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.284+0000 7f1e0a240640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:32.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.687+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de800a660 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:32.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.687+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1de80102a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:32.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.687+0000 7f1e0a240640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 msgr2=0x7f1de8013110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:32.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.687+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80102a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:32.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:32.687+0000 7f1e0a240640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:33.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:33.491+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de800a660 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:33.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:33.491+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1de80102a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:33.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:33.491+0000 7f1e0a240640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 msgr2=0x7f1de8013110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:33.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:33.491+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80102a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:33.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:33.491+0000 7f1e0a240640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:35.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:35.079+0000 7f1dfb7fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 msgr2=0x7f1de8013110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:35.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:35.079+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:35.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:35.079+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1de80102a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:35.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:35.079+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80102a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:35.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:35.079+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de800a660 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:35.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:35.079+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800da30 0x7f1de8009090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:35.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:35.079+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415ad50 con 0x7f1de8007ca0 2023-12-04T05:50:35.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:35.079+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de800da30 2023-12-04T05:50:35.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:35.079+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800da30 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:35.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:35.079+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8007ca0 legacy=0x7f1de800d280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:35.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:35.079+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800da30 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:35.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:35.283+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800da30 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:35.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:35.283+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800da30 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:35.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:35.283+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8007ca0 legacy=0x7f1de800d280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:35.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:35.687+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800da30 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:35.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:35.687+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800da30 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:35.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:35.687+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8007ca0 legacy=0x7f1de800d280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:36.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:36.491+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800da30 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:36.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:36.491+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800da30 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:36.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:36.491+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8007ca0 legacy=0x7f1de800d280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:38.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.079+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800da30 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:38.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.079+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800da30 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:38.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.079+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8007ca0 legacy=0x7f1de800d280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:38.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.079+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:38.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.079+0000 7f1e0a240640 1 -- reap_dead start 2023-12-04T05:50:38.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.079+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80095d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:38.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.079+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800da30 0x7f1de8009090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:38.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.079+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de800da30 2023-12-04T05:50:38.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.079+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de80136f0 con 0x7f1e04154020 2023-12-04T05:50:38.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.079+0000 7f1dfb7fe640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de8009b10 con 0x7f1de8010d20 2023-12-04T05:50:38.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.079+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800da30 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:38.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.079+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800da30 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:38.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.079+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1de80095d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:38.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.079+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 msgr2=0x7f1de8013110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:38.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.079+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80095d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:38.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.079+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:38.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.283+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800da30 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:38.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.283+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800da30 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:38.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.283+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 msgr2=0x7f1de8013110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:38.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.283+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1de80095d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:38.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.283+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:38.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.283+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80095d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:38.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.687+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800da30 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:38.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.687+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800da30 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:38.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.687+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 msgr2=0x7f1de8013110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:38.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.687+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1de80095d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:38.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.687+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80095d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:38.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:38.687+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:39.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:39.491+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800da30 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:39.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:39.491+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800da30 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:39.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:39.491+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1de80095d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:39.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:39.491+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 msgr2=0x7f1de8013110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:39.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:39.491+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:39.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:39.491+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80095d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:41.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:41.079+0000 7f1dfb7fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 msgr2=0x7f1de8013110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:41.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:41.079+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010d20 0x7f1de8013110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:41.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:41.079+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 msgr2=0x7f1de80095d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:41.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:41.079+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1e04154020 0x7f1de80095d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:41.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:41.079+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800da30 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:41.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:41.079+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800da30 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:41.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:41.079+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de800ac20 con 0x7f1de8008cb0 2023-12-04T05:50:41.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:41.079+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415ad50 con 0x7f1de800fef0 2023-12-04T05:50:41.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:41.079+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8008cb0 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:41.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:41.079+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de800fef0 legacy=0x7f1de800d280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:41.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:41.283+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8008cb0 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:41.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:41.283+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de800fef0 legacy=0x7f1de800d280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:41.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:41.687+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8008cb0 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:41.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:41.687+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de800fef0 legacy=0x7f1de800d280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:42.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:42.491+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8008cb0 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:42.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:42.491+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de800fef0 legacy=0x7f1de800d280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:44.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:44.079+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de800fef0 legacy=0x7f1de800d280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:44.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:44.079+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8008cb0 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:44.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:44.079+0000 7f1e0a240640 1 -- reap_dead start 2023-12-04T05:50:44.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:44.079+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de80094a0 0x7f1de800d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:44.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:44.079+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415d710 con 0x7f1de8009090 2023-12-04T05:50:44.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:44.079+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de800ac20 con 0x7f1de8010340 2023-12-04T05:50:44.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:44.079+0000 7f1dfb7fe640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de80094a0 2023-12-04T05:50:44.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:44.079+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8009090 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:44.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:44.079+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8010340 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:44.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:44.079+0000 7f1e0a240640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de80094a0 msgr2=0x7f1de800d280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:44.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:44.079+0000 7f1e0a240640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de80094a0 0x7f1de800d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:44.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:44.283+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8009090 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:44.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:44.283+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8010340 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:44.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:44.283+0000 7f1e0a240640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de80094a0 msgr2=0x7f1de800d280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:44.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:44.283+0000 7f1e0a240640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de80094a0 0x7f1de800d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:44.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:44.687+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8009090 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:44.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:44.687+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8010340 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:44.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:44.687+0000 7f1e0a240640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de80094a0 msgr2=0x7f1de800d280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:44.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:44.687+0000 7f1e0a240640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de80094a0 0x7f1de800d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:45.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:45.491+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8009090 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:45.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:45.491+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8010340 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:45.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:45.491+0000 7f1e0a240640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de80094a0 msgr2=0x7f1de800d280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:45.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:45.491+0000 7f1e0a240640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de80094a0 0x7f1de800d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:47.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:47.079+0000 7f1dfb7fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de80094a0 msgr2=0x7f1de800d280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:47.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:47.079+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de80094a0 0x7f1de800d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:47.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:47.079+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8010340 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:47.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:47.079+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8009090 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:47.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:47.079+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800d7c0 0x7f1de8013920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:47.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:47.079+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8013e60 0x7f1de8016250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:47.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:47.079+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de800d7c0 2023-12-04T05:50:47.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:47.079+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de80136f0 con 0x7f1de8013e60 2023-12-04T05:50:47.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:47.079+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800d7c0 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:47.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:47.079+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800d7c0 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:47.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:47.079+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8013e60 msgr2=0x7f1de8016250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:47.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:47.079+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8013e60 0x7f1de8016250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:47.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:47.283+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800d7c0 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:47.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:47.283+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800d7c0 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:47.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:47.283+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8013e60 msgr2=0x7f1de8016250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:47.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:47.283+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8013e60 0x7f1de8016250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:47.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:47.687+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800d7c0 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:47.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:47.687+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800d7c0 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:47.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:47.687+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8013e60 msgr2=0x7f1de8016250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:47.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:47.687+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8013e60 0x7f1de8016250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:48.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:48.491+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800d7c0 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:48.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:48.491+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800d7c0 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:48.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:48.491+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8013e60 msgr2=0x7f1de8016250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:48.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:48.491+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8013e60 0x7f1de8016250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:50.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:50.079+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8013e60 msgr2=0x7f1de8016250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:50.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:50.079+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8013e60 0x7f1de8016250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:50.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:50.079+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800d7c0 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:50.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:50.079+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800d7c0 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:50.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:50.079+0000 7f1e0a240640 1 -- reap_dead start 2023-12-04T05:50:50.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:50.079+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de80094a0 0x7f1de8013920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:50.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:50.079+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415ad50 con 0x7f1de8010310 2023-12-04T05:50:50.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:50.079+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415d710 con 0x7f1de8009090 2023-12-04T05:50:50.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:50.079+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de80094a0 2023-12-04T05:50:50.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:50.079+0000 7f1e0a240640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:50.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:50.083+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8009090 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:50.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:50.083+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de80094a0 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:50.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:50.083+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de80094a0 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:50.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:50.283+0000 7f1e0a240640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:50.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:50.287+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de80094a0 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:50.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:50.287+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8009090 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:50.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:50.287+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de80094a0 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:50.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:50.687+0000 7f1e0a240640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:50.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:50.691+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8009090 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:50.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:50.691+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de80094a0 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:50.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:50.691+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de80094a0 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:51.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:51.491+0000 7f1e0a240640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:51.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:51.495+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de80094a0 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:51.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:51.495+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8009090 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:51.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:51.495+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de80094a0 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:53.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:53.083+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de80094a0 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:53.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:53.083+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de80094a0 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:53.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:53.083+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8009090 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:53.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:53.083+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:53.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:53.083+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800f390 0x7f1de800f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:53.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:53.083+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8014240 0x7f1de8016630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:53.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:53.083+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de800f390 2023-12-04T05:50:53.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:53.083+0000 7f1dfb7fe640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80136f0 con 0x7f1de8014240 2023-12-04T05:50:53.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:53.083+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800f390 msgr2=0x7f1de800f7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:53.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:53.083+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800f390 0x7f1de800f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:53.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:53.083+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8014240 msgr2=0x7f1de8016630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:53.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:53.083+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8014240 0x7f1de8016630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:53.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:53.287+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800f390 msgr2=0x7f1de800f7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:53.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:53.287+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800f390 0x7f1de800f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:53.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:53.287+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8014240 msgr2=0x7f1de8016630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:53.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:53.287+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8014240 0x7f1de8016630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:53.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:53.691+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800f390 msgr2=0x7f1de800f7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:53.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:53.691+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800f390 0x7f1de800f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:53.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:53.691+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8014240 msgr2=0x7f1de8016630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:53.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:53.691+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8014240 0x7f1de8016630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:54.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:54.495+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800f390 msgr2=0x7f1de800f7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:54.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:54.495+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800f390 0x7f1de800f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:54.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:54.495+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8014240 msgr2=0x7f1de8016630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:54.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:54.495+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8014240 0x7f1de8016630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:56.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.083+0000 7f1dfb7fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8014240 msgr2=0x7f1de8016630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:56.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.083+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8014240 0x7f1de8016630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:56.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.083+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de800f390 msgr2=0x7f1de800f7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:56.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.083+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de800f390 0x7f1de800f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:56.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.083+0000 7f1e0a240640 1 -- reap_dead start 2023-12-04T05:50:56.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.083+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009090 0x7f1de8016630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:56.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.083+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de80094a0 0x7f1de8013920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:56.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.083+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de800ac20 con 0x7f1de8010310 2023-12-04T05:50:56.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.083+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8009090 2023-12-04T05:50:56.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.083+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de80136f0 con 0x7f1de80094a0 2023-12-04T05:50:56.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.083+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de8016630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:56.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.083+0000 7f1e0a240640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:56.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.083+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009090 0x7f1de8016630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:56.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.083+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de80094a0 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:56.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.083+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de80094a0 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:56.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.287+0000 7f1e0a240640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:56.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.287+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de8016630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:56.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.287+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009090 0x7f1de8016630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:56.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.287+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de80094a0 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:56.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.287+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de80094a0 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:56.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.691+0000 7f1e0a240640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:56.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.691+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de8016630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:56.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.691+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009090 0x7f1de8016630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:56.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.691+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de80094a0 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:56.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:56.691+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de80094a0 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:50:57.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:57.495+0000 7f1e0a240640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:50:57.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:57.495+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de8016630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:50:57.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:57.495+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009090 0x7f1de8016630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:57.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:57.495+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de80094a0 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:50:57.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:57.495+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de80094a0 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:50:59.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:59.083+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de80094a0 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:59.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:59.083+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de80094a0 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:59.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:59.083+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de8016630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:59.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:59.083+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009090 0x7f1de8016630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:50:59.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:59.083+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:50:59.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:59.083+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de800f770 0x7f1de8005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:50:59.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:59.083+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415ad50 con 0x7f1de8006060 2023-12-04T05:50:59.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:59.083+0000 7f1dfb7fe640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de800f770 2023-12-04T05:50:59.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:59.083+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8006060 legacy=0x7f1de8006470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:59.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:59.083+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de800f770 msgr2=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:59.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:59.083+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de800f770 0x7f1de8005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:50:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:59.287+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8006060 legacy=0x7f1de8006470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:59.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:59.287+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de800f770 msgr2=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:59.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:59.287+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de800f770 0x7f1de8005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:50:59.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:59.691+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de800f770 msgr2=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:50:59.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:59.691+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8006060 legacy=0x7f1de8006470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:50:59.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:50:59.691+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de800f770 0x7f1de8005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:00.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:00.495+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8006060 legacy=0x7f1de8006470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:00.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:00.495+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de800f770 msgr2=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:00.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:00.495+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de800f770 0x7f1de8005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:02.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.083+0000 7f1dfb7fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de800f770 msgr2=0x7f1de8005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:02.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.083+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de800f770 0x7f1de8005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:02.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.083+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8006060 legacy=0x7f1de8006470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:02.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.083+0000 7f1e0a240640 1 -- reap_dead start 2023-12-04T05:51:02.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.083+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 0x7f1de8013920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.083+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de8006c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.083+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de800ac20 con 0x7f1de80094a0 2023-12-04T05:51:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.083+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8009090 2023-12-04T05:51:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.083+0000 7f1dfb7fe640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80136f0 con 0x7f1de8010310 2023-12-04T05:51:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.083+0000 7f1e0a240640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 msgr2=0x7f1de8006c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.083+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de80094a0 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.083+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.083+0000 7f1e0a240640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de8006c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:02.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.083+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:02.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.287+0000 7f1e0a240640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 msgr2=0x7f1de8006c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:02.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.287+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de80094a0 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:02.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.287+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:02.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.287+0000 7f1e0a240640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de8006c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:02.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.287+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:02.331 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:51:02.334 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:51:02.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.691+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de80094a0 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:02.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.691+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:02.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.691+0000 7f1e0a240640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 msgr2=0x7f1de8006c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:02.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.691+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:02.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:02.691+0000 7f1e0a240640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de8006c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:03.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:03.495+0000 7f1e0a240640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 msgr2=0x7f1de8006c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:03.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:03.495+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:03.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:03.495+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de80094a0 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:03.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:03.495+0000 7f1e0a240640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de8006c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:03.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:03.495+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:05.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:05.083+0000 7f1dfb7fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 msgr2=0x7f1de8006c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:05.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:05.083+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de8006c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:05.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:05.083+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:05.083+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:05.083+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de80094a0 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:05.083+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8006640 0x7f1de8017640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:05.083+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415ad50 con 0x7f1de8005b20 2023-12-04T05:51:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:05.083+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8006640 2023-12-04T05:51:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:05.083+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8005b20 legacy=0x7f1de8005f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:05.083+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8006640 msgr2=0x7f1de8017640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:05.083+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8006640 0x7f1de8017640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:05.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:05.287+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8006640 msgr2=0x7f1de8017640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:05.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:05.287+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8005b20 legacy=0x7f1de8005f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:05.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:05.287+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8006640 0x7f1de8017640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:05.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:05.691+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8005b20 legacy=0x7f1de8005f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:05.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:05.691+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8006640 msgr2=0x7f1de8017640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:05.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:05.691+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8006640 0x7f1de8017640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:06.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:06.495+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8006640 msgr2=0x7f1de8017640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:06.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:06.495+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8005b20 legacy=0x7f1de8005f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:06.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:06.495+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8006640 0x7f1de8017640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:08.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.083+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8006640 msgr2=0x7f1de8017640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:08.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.083+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8006640 0x7f1de8017640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:08.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.083+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8005b20 legacy=0x7f1de8005f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:08.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.083+0000 7f1e0a240640 1 -- reap_dead start 2023-12-04T05:51:08.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.083+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 0x7f1de8013920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:08.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.083+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de80094a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:08.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.083+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de800ac20 con 0x7f1de8006b80 2023-12-04T05:51:08.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.083+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8009090 2023-12-04T05:51:08.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.083+0000 7f1dfb7fe640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80136f0 con 0x7f1de8010310 2023-12-04T05:51:08.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.083+0000 7f1e0a240640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8006b80 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:08.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.083+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 msgr2=0x7f1de80094a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:08.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.083+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:08.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.083+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:08.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.083+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de80094a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:08.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.287+0000 7f1e0a240640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8006b80 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:08.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.287+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:08.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.287+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 msgr2=0x7f1de80094a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:08.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.287+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:08.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.287+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de80094a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:08.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.691+0000 7f1e0a240640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8006b80 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:08.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.691+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:08.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.691+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 msgr2=0x7f1de80094a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:08.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.691+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:08.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:08.691+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de80094a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:09.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:09.495+0000 7f1e0a240640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8006b80 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:09.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:09.495+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:09.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:09.495+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 msgr2=0x7f1de80094a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:09.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:09.495+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:09.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:09.495+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de80094a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:11.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:11.083+0000 7f1dfb7fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 msgr2=0x7f1de80094a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:11.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:11.083+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de80094a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:11.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:11.083+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de8013920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:11.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:11.083+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 0x7f1de8013920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:11.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:11.083+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8006b80 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:11.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:11.083+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8005b20 0x7f1de8005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:11.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:11.083+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415ad50 con 0x7f1de8006470 2023-12-04T05:51:11.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:11.083+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8005b20 2023-12-04T05:51:11.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:11.083+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8005b20 msgr2=0x7f1de8005f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:11.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:11.083+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8005b20 0x7f1de8005f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:11.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:11.083+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8006470 legacy=0x7f1de8016630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:11.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:11.287+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8005b20 msgr2=0x7f1de8005f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:11.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:11.287+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8005b20 0x7f1de8005f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:11.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:11.287+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8006470 legacy=0x7f1de8016630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:11.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:11.691+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8005b20 msgr2=0x7f1de8005f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:11.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:11.691+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8005b20 0x7f1de8005f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:11.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:11.691+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8006470 legacy=0x7f1de8016630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:12.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:12.495+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8005b20 msgr2=0x7f1de8005f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:12.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:12.495+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8005b20 0x7f1de8005f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:12.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:12.495+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8006470 legacy=0x7f1de8016630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:14.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.083+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8005b20 msgr2=0x7f1de8005f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:14.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.083+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8005b20 0x7f1de8005f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:14.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.083+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8006470 legacy=0x7f1de8016630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:14.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.083+0000 7f1e0a240640 1 -- reap_dead start 2023-12-04T05:51:14.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.083+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 0x7f1de80094a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:14.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.083+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009090 0x7f1de80138d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:14.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.083+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de800ac20 con 0x7f1de8010310 2023-12-04T05:51:14.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.083+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8009090 2023-12-04T05:51:14.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.083+0000 7f1dfb7fe640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80136f0 con 0x7f1de8006b80 2023-12-04T05:51:14.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.083+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de80138d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:14.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.083+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009090 0x7f1de80138d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:14.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.083+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 msgr2=0x7f1de80094a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:14.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.083+0000 7f1e0a240640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:14.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.083+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 0x7f1de80094a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:14.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.287+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de80138d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:14.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.287+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009090 0x7f1de80138d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:14.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.287+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 msgr2=0x7f1de80094a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:14.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.287+0000 7f1e0a240640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:14.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.287+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 0x7f1de80094a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:14.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.691+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de80138d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:14.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.691+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009090 0x7f1de80138d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:14.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.691+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 msgr2=0x7f1de80094a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:14.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.691+0000 7f1e0a240640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:14.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:14.691+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 0x7f1de80094a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:15.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:15.495+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de80138d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:15.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:15.495+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009090 0x7f1de80138d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:15.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:15.495+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 msgr2=0x7f1de80094a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:15.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:15.495+0000 7f1e0a240640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:15.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:15.495+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 0x7f1de80094a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:17.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:17.083+0000 7f1dfb7fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 msgr2=0x7f1de80094a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:17.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:17.083+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 0x7f1de80094a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:17.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:17.083+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de80138d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:17.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:17.083+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009090 0x7f1de80138d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:17.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:17.083+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:17.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:17.083+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8005b20 0x7f1de8005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:17.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:17.083+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415ad50 con 0x7f1de8014620 2023-12-04T05:51:17.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:17.083+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8005b20 2023-12-04T05:51:17.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:17.083+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8014620 legacy=0x7f1de8016a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:17.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:17.083+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8005b20 msgr2=0x7f1de8005f00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:17.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:17.083+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8005b20 0x7f1de8005f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:17.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:17.287+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8014620 legacy=0x7f1de8016a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:17.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:17.287+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8005b20 msgr2=0x7f1de8005f00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:17.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:17.287+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8005b20 0x7f1de8005f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:17.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:17.691+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8014620 legacy=0x7f1de8016a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:17.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:17.691+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8005b20 msgr2=0x7f1de8005f00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:17.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:17.691+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8005b20 0x7f1de8005f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:18.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:18.495+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8014620 legacy=0x7f1de8016a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:18.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:18.495+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8005b20 msgr2=0x7f1de8005f00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:18.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:18.495+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8005b20 0x7f1de8005f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:20.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.083+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8005b20 msgr2=0x7f1de8005f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:20.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.083+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8005b20 0x7f1de8005f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:20.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.083+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8014620 legacy=0x7f1de8016a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:20.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.083+0000 7f1e0a240640 1 -- reap_dead start 2023-12-04T05:51:20.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.083+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8009090 0x7f1de80094a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:20.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.083+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 0x7f1de80138d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:20.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.083+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de800ac20 con 0x7f1de8006b80 2023-12-04T05:51:20.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.083+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8010310 2023-12-04T05:51:20.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.083+0000 7f1dfb7fe640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80136f0 con 0x7f1de8009090 2023-12-04T05:51:20.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.083+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8006b80 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:20.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.083+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8009090 msgr2=0x7f1de80094a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:20.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.083+0000 7f1e0a240640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 msgr2=0x7f1de80138d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:20.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.083+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8009090 0x7f1de80094a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:20.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.083+0000 7f1e0a240640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 0x7f1de80138d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:20.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.287+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8006b80 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:20.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.287+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8009090 msgr2=0x7f1de80094a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:20.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.287+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8009090 0x7f1de80094a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:20.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.287+0000 7f1e0a240640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 msgr2=0x7f1de80138d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:20.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.287+0000 7f1e0a240640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 0x7f1de80138d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:20.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.691+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8006b80 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:20.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.691+0000 7f1e0a240640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 msgr2=0x7f1de80138d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:20.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.691+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8009090 msgr2=0x7f1de80094a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:20.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.691+0000 7f1e0a240640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 0x7f1de80138d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:20.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:20.691+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8009090 0x7f1de80094a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:21.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:21.495+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8006b80 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:21.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:21.495+0000 7f1e0a240640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 msgr2=0x7f1de80138d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:21.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:21.495+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8009090 msgr2=0x7f1de80094a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:21.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:21.495+0000 7f1e0a240640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 0x7f1de80138d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:21.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:21.495+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8009090 0x7f1de80094a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:23.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:23.087+0000 7f1dfb7fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8009090 msgr2=0x7f1de80094a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:23.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:23.087+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8009090 0x7f1de80094a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:23.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:23.087+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 msgr2=0x7f1de80138d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:23.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:23.087+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 0x7f1de80138d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:23.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:23.087+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8006b80 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:23.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:23.087+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8005f00 0x7f1de8017d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:23.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:23.087+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415ad50 con 0x7f1de8005b20 2023-12-04T05:51:23.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:23.087+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8005f00 2023-12-04T05:51:23.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:23.087+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8005f00 msgr2=0x7f1de8017d50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:23.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:23.087+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8005f00 0x7f1de8017d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:23.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:23.087+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8005b20 legacy=0x7f1de8014620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:23.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:23.291+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8005f00 msgr2=0x7f1de8017d50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:23.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:23.291+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8005f00 0x7f1de8017d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:23.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:23.291+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8005b20 legacy=0x7f1de8014620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:23.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:23.695+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8005f00 msgr2=0x7f1de8017d50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:23.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:23.695+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8005f00 0x7f1de8017d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:23.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:23.695+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8005b20 legacy=0x7f1de8014620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:24.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:24.499+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8005f00 msgr2=0x7f1de8017d50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:24.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:24.499+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8005f00 0x7f1de8017d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:24.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:24.499+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8005b20 legacy=0x7f1de8014620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:26.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.087+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8005f00 msgr2=0x7f1de8017d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:26.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.087+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8005f00 0x7f1de8017d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:26.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.087+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8005b20 legacy=0x7f1de8014620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:26.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.087+0000 7f1e0a240640 1 -- reap_dead start 2023-12-04T05:51:26.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.087+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 0x7f1de80138d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:26.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.087+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de8006b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:26.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.087+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009470 0x7f1de8012d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:26.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.087+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8009470 2023-12-04T05:51:26.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.087+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de80136f0 con 0x7f1de8009090 2023-12-04T05:51:26.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.087+0000 7f1dfb7fe640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de8009b10 con 0x7f1de8010310 2023-12-04T05:51:26.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.087+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009470 msgr2=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:26.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.087+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009470 0x7f1de8012d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:26.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.087+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 msgr2=0x7f1de8006b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:26.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.087+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de80138d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:26.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.087+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 0x7f1de80138d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:26.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.087+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de8006b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:26.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.291+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009470 msgr2=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:26.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.291+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009470 0x7f1de8012d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:26.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.291+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de80138d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:26.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.291+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 msgr2=0x7f1de8006b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:26.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.291+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 0x7f1de80138d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:26.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.291+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de8006b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:26.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.695+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009470 msgr2=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:26.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.695+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009470 0x7f1de8012d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:26.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.695+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de80138d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:26.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.695+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 msgr2=0x7f1de8006b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:26.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.695+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 0x7f1de80138d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:26.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:26.695+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de8006b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:27.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:27.499+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009470 msgr2=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:27.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:27.499+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009470 0x7f1de8012d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:27.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:27.499+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 msgr2=0x7f1de8006b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:27.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:27.499+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de80138d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:27.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:27.499+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de8006b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:27.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:27.499+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 0x7f1de80138d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:29.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:29.087+0000 7f1dfb7fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 msgr2=0x7f1de8006b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:29.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:29.087+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de8006b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:29.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:29.087+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 msgr2=0x7f1de80138d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:29.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:29.087+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8009090 0x7f1de80138d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:29.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:29.087+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009470 msgr2=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:29.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:29.087+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8009470 0x7f1de8012d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:29.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:29.087+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de800ac20 con 0x7f1de8005f30 2023-12-04T05:51:29.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:29.087+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415ad50 con 0x7f1de8005b20 2023-12-04T05:51:29.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:29.087+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8005f30 legacy=0x7f1de8008320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:29.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:29.087+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8005b20 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:29.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:29.291+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8005f30 legacy=0x7f1de8008320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:29.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:29.291+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8005b20 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:29.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:29.695+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8005f30 legacy=0x7f1de8008320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:29.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:29.695+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8005b20 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:30.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:30.499+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8005f30 legacy=0x7f1de8008320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:30.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:30.499+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8005b20 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:32.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:32.087+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8005b20 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:32.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:32.087+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8005f30 legacy=0x7f1de8008320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:32.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:32.087+0000 7f1e0a240640 1 -- reap_dead start 2023-12-04T05:51:32.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:32.087+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8013830 0x7f1de800e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:32.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:32.087+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415d710 con 0x7f1de8010310 2023-12-04T05:51:32.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:32.087+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de800ac20 con 0x7f1de8006b80 2023-12-04T05:51:32.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:32.087+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8013830 2023-12-04T05:51:32.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:32.087+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:32.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:32.087+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8013830 msgr2=0x7f1de800e270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:32.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:32.087+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8013830 0x7f1de800e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:32.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:32.087+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8006b80 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:32.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:32.291+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8013830 msgr2=0x7f1de800e270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:32.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:32.291+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:32.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:32.291+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8013830 0x7f1de800e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:32.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:32.291+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8006b80 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:32.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:51:32.486 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:51:32.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:32.695+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8013830 msgr2=0x7f1de800e270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:32.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:32.695+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:32.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:32.695+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8013830 0x7f1de800e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:32.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:32.695+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8006b80 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:33.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:33.499+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8013830 msgr2=0x7f1de800e270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:33.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:33.499+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:33.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:33.499+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8013830 0x7f1de800e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:33.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:33.499+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8006b80 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:35.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:35.087+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8013830 msgr2=0x7f1de800e270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:35.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:35.087+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8013830 0x7f1de800e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:35.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:35.087+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8006b80 legacy=0x7f1de8012d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:35.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:35.087+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:35.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:35.087+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de800e7b0 0x7f1de800ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:35.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:35.087+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8016630 0x7f1de8018a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:35.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:35.087+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de800e7b0 2023-12-04T05:51:35.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:35.087+0000 7f1dfb7fe640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80136f0 con 0x7f1de8016630 2023-12-04T05:51:35.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:35.087+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de800e7b0 msgr2=0x7f1de800ebc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:35.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:35.087+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8016630 msgr2=0x7f1de8018a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:35.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:35.087+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de800e7b0 0x7f1de800ebc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:35.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:35.087+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8016630 0x7f1de8018a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:35.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:35.291+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de800e7b0 msgr2=0x7f1de800ebc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:35.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:35.291+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8016630 msgr2=0x7f1de8018a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:35.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:35.291+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de800e7b0 0x7f1de800ebc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:35.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:35.291+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8016630 0x7f1de8018a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:35.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:35.695+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de800e7b0 msgr2=0x7f1de800ebc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:35.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:35.695+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8016630 msgr2=0x7f1de8018a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:35.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:35.695+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de800e7b0 0x7f1de800ebc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:35.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:35.695+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8016630 0x7f1de8018a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:36.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:36.499+0000 7f1e0923e640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8016630 msgr2=0x7f1de8018a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:36.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:36.499+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de800e7b0 msgr2=0x7f1de800ebc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:36.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:36.499+0000 7f1e0923e640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8016630 0x7f1de8018a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:36.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:36.499+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de800e7b0 0x7f1de800ebc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:38.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:38.087+0000 7f1dfb7fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8016630 msgr2=0x7f1de8018a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:38.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:38.087+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8016630 0x7f1de8018a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:38.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:38.087+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de800e7b0 msgr2=0x7f1de800ebc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:38.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:38.087+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de800e7b0 0x7f1de800ebc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:38.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:38.087+0000 7f1e0a240640 1 -- reap_dead start 2023-12-04T05:51:38.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:38.087+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 0x7f1de8012440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:38.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:38.087+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415ad50 con 0x7f1de8013830 2023-12-04T05:51:38.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:38.087+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415d710 con 0x7f1de8010310 2023-12-04T05:51:38.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:38.087+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8006b80 2023-12-04T05:51:38.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:38.087+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8013830 legacy=0x7f1de8010d20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:38.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:38.087+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:38.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:38.087+0000 7f1e0a240640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 msgr2=0x7f1de8012440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:38.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:38.087+0000 7f1e0a240640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 0x7f1de8012440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:38.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:38.291+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8013830 legacy=0x7f1de8010d20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:38.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:38.291+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:38.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:38.291+0000 7f1e0a240640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 msgr2=0x7f1de8012440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:38.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:38.291+0000 7f1e0a240640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 0x7f1de8012440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:38.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:38.695+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8013830 legacy=0x7f1de8010d20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:38.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:38.695+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:38.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:38.695+0000 7f1e0a240640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 msgr2=0x7f1de8012440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:38.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:38.695+0000 7f1e0a240640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 0x7f1de8012440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:39.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:39.499+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8013830 legacy=0x7f1de8010d20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:39.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:39.499+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:39.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:39.499+0000 7f1e0a240640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 msgr2=0x7f1de8012440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:39.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:39.499+0000 7f1e0a240640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 0x7f1de8012440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:41.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:41.087+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 msgr2=0x7f1de8012440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:41.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:41.087+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 0x7f1de8012440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:41.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:41.087+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:41.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:41.087+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8013830 legacy=0x7f1de8010d20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:41.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:41.087+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8012980 0x7f1de8012d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:41.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:41.087+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8016a10 0x7f1de8018e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:41.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:41.087+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8016a10 2023-12-04T05:51:41.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:41.087+0000 7f1dfb7fe640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80136f0 con 0x7f1de8012980 2023-12-04T05:51:41.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:41.087+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8016a10 msgr2=0x7f1de8018e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:41.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:41.087+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8016a10 0x7f1de8018e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:41.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:41.087+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8012980 msgr2=0x7f1de8012d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:41.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:41.087+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8012980 0x7f1de8012d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:41.291+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8016a10 msgr2=0x7f1de8018e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:41.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:41.291+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8016a10 0x7f1de8018e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:41.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:41.291+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8012980 msgr2=0x7f1de8012d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:41.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:41.291+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8012980 0x7f1de8012d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:41.695+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8016a10 msgr2=0x7f1de8018e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:41.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:41.695+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8016a10 0x7f1de8018e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:41.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:41.695+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8012980 msgr2=0x7f1de8012d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:41.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:41.695+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8012980 0x7f1de8012d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:42.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:42.499+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8016a10 msgr2=0x7f1de8018e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:42.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:42.499+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8016a10 0x7f1de8018e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:42.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:42.499+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8012980 msgr2=0x7f1de8012d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:42.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:42.499+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8012980 0x7f1de8012d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:44.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.087+0000 7f1dfb7fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8012980 msgr2=0x7f1de8012d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:44.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.087+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8012980 0x7f1de8012d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:44.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.087+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8016a10 msgr2=0x7f1de8018e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:44.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.087+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8016a10 0x7f1de8018e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:44.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.087+0000 7f1e0a240640 1 -- reap_dead start 2023-12-04T05:51:44.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.087+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 0x7f1de8012d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:44.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.087+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 0x7f1de8009090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:44.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.087+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de800ac20 con 0x7f1de8013830 2023-12-04T05:51:44.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.087+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8010310 2023-12-04T05:51:44.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.087+0000 7f1dfb7fe640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80136f0 con 0x7f1de8006b80 2023-12-04T05:51:44.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.087+0000 7f1e0a240640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8013830 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:44.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.087+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 msgr2=0x7f1de8012d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:44.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.087+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:44.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.087+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 0x7f1de8012d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:44.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.087+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:44.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.291+0000 7f1e0a240640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8013830 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:44.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.291+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:44.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.291+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 msgr2=0x7f1de8012d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:44.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.291+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:44.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.291+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 0x7f1de8012d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:44.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.695+0000 7f1e0a240640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8013830 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.695+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 msgr2=0x7f1de8012d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.695+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.695+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 0x7f1de8012d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:44.695+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:45.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:45.499+0000 7f1e0a240640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8013830 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:45.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:45.499+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 msgr2=0x7f1de8012d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:45.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:45.499+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:45.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:45.499+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:45.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:45.499+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 0x7f1de8012d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:47.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:47.087+0000 7f1dfb7fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 msgr2=0x7f1de8012d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:47.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:47.087+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 0x7f1de8012d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:47.087+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:47.087+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:47.087+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8013830 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:47.087+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8008320 0x7f1de8008700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:47.087+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415ad50 con 0x7f1de80095d0 2023-12-04T05:51:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:47.087+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8008320 2023-12-04T05:51:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:47.087+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8008320 msgr2=0x7f1de8008700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:47.087+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8008320 0x7f1de8008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:47.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:47.087+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de80095d0 legacy=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:47.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:47.291+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8008320 msgr2=0x7f1de8008700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:47.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:47.291+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8008320 0x7f1de8008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:47.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:47.291+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de80095d0 legacy=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:47.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:47.695+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8008320 msgr2=0x7f1de8008700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:47.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:47.695+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8008320 0x7f1de8008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:47.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:47.695+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de80095d0 legacy=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:48.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:48.499+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8008320 msgr2=0x7f1de8008700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:48.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:48.499+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8008320 0x7f1de8008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:48.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:48.499+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de80095d0 legacy=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:50.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.087+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8008320 msgr2=0x7f1de8008700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:50.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.087+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8008320 0x7f1de8008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:50.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.087+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de80095d0 legacy=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:50.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.087+0000 7f1e0a240640 1 -- reap_dead start 2023-12-04T05:51:50.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.087+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 0x7f1de8012d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:50.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.087+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 0x7f1de8009090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:50.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.087+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8013830 0x7f1de8007c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:50.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.087+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8013830 2023-12-04T05:51:50.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.087+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de80136f0 con 0x7f1de8010310 2023-12-04T05:51:50.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.087+0000 7f1dfb7fe640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de8009b10 con 0x7f1de8006b80 2023-12-04T05:51:50.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.087+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8013830 msgr2=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:50.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.087+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8013830 0x7f1de8007c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:50.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.087+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 msgr2=0x7f1de8012d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:50.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.087+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:50.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.087+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 0x7f1de8012d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:50.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.087+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:50.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.291+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8013830 msgr2=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:50.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.291+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8013830 0x7f1de8007c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:50.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.291+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 msgr2=0x7f1de8012d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:50.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.291+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:50.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.291+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:50.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.291+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 0x7f1de8012d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:50.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.695+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8013830 msgr2=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:50.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.695+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8013830 0x7f1de8007c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:50.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.695+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 msgr2=0x7f1de8012d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:50.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.695+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:50.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.695+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:50.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:50.695+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 0x7f1de8012d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:51.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:51.499+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8013830 msgr2=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:51.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:51.499+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8013830 0x7f1de8007c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:51.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:51.499+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 msgr2=0x7f1de8012d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:51.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:51.499+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:51.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:51.499+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 0x7f1de8012d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:51.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:51.499+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:53.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:53.090+0000 7f1dfb7fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 msgr2=0x7f1de8012d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:53.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:53.090+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8006b80 0x7f1de8012d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:53.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:53.090+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:53.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:53.090+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8010310 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:53.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:53.090+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8013830 msgr2=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:53.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:53.090+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8013830 0x7f1de8007c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:53.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:53.090+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de800ac20 con 0x7f1de8008860 2023-12-04T05:51:53.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:53.090+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415ad50 con 0x7f1de80095d0 2023-12-04T05:51:53.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:53.090+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8008860 legacy=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:53.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:53.090+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de80095d0 legacy=0x7f1de8008150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:53.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:53.294+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8008860 legacy=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:53.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:53.294+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de80095d0 legacy=0x7f1de8008150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:53.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:53.698+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8008860 legacy=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:53.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:53.698+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de80095d0 legacy=0x7f1de8008150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:54.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:54.502+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8008860 legacy=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:54.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:54.502+0000 7f1e09a3f640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de80095d0 legacy=0x7f1de8008150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:56.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:56.090+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de80095d0 legacy=0x7f1de8008150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:56.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:56.090+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8008860 legacy=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:56.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:56.090+0000 7f1e0a240640 1 -- reap_dead start 2023-12-04T05:51:56.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:56.090+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 0x7f1de8009090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:56.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:56.090+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415d710 con 0x7f1de8013830 2023-12-04T05:51:56.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:56.090+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de800ac20 con 0x7f1de8010310 2023-12-04T05:51:56.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:56.090+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8006b80 2023-12-04T05:51:56.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:56.090+0000 7f1e0a240640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8013830 legacy=0x7f1de8008320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:56.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:56.090+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:56.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:56.090+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:56.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:56.090+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:56.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:56.294+0000 7f1e0a240640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8013830 legacy=0x7f1de8008320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:56.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:56.294+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:56.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:56.294+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:56.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:56.294+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:56.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:56.698+0000 7f1e0a240640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8013830 legacy=0x7f1de8008320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:56.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:56.698+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:56.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:56.698+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:56.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:56.698+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:57.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:57.502+0000 7f1e0a240640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8013830 legacy=0x7f1de8008320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:51:57.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:57.502+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:51:57.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:57.502+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:51:57.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:57.502+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:51:59.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:59.090+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:59.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:59.090+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:51:59.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:59.090+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8010310 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:59.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:59.090+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8013830 legacy=0x7f1de8008320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:51:59.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:59.090+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8012d60 0x7f1de8011d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:59.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:59.090+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8012270 0x7f1de800e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:51:59.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:59.090+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8012270 2023-12-04T05:51:59.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:59.090+0000 7f1dfb7fe640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80136f0 con 0x7f1de8012d60 2023-12-04T05:51:59.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:59.090+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8012270 msgr2=0x7f1de800e270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:59.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:59.090+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8012270 0x7f1de800e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:59.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:59.090+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8012d60 msgr2=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:59.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:59.090+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8012d60 0x7f1de8011d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:51:59.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:59.294+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8012270 msgr2=0x7f1de800e270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:59.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:59.294+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8012270 0x7f1de800e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:59.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:59.294+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8012d60 msgr2=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:59.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:59.294+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8012d60 0x7f1de8011d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:51:59.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:59.698+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8012270 msgr2=0x7f1de800e270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:51:59.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:59.698+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8012270 0x7f1de800e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:51:59.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:59.698+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8012d60 msgr2=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:51:59.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:51:59.698+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8012d60 0x7f1de8011d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:52:00.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:00.502+0000 7f1e0923e640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8012270 msgr2=0x7f1de800e270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:52:00.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:00.502+0000 7f1e0923e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8012270 0x7f1de800e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:52:00.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:00.502+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8012d60 msgr2=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:52:00.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:00.502+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8012d60 0x7f1de8011d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:52:02.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.090+0000 7f1dfb7fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8012d60 msgr2=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:52:02.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.090+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8012d60 0x7f1de8011d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:52:02.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.090+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8012270 msgr2=0x7f1de800e270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:52:02.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.090+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8012270 0x7f1de800e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:52:02.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.090+0000 7f1e0a240640 1 -- reap_dead start 2023-12-04T05:52:02.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.090+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 0x7f1de8009090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:52:02.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.090+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8010310 0x7f1de8008320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:52:02.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.090+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415ad50 con 0x7f1de8012d60 2023-12-04T05:52:02.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.090+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8010310 2023-12-04T05:52:02.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.090+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de80136f0 con 0x7f1de8006b80 2023-12-04T05:52:02.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.090+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8012d60 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:52:02.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.090+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8010310 msgr2=0x7f1de8008320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:52:02.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.090+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8010310 0x7f1de8008320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:52:02.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.090+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:52:02.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.090+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:52:02.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.294+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8010310 msgr2=0x7f1de8008320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:52:02.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.294+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8012d60 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:52:02.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.294+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8010310 0x7f1de8008320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:52:02.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.294+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:52:02.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.294+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:52:02.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:52:02.599 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:52:02.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.698+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8010310 msgr2=0x7f1de8008320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:52:02.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.698+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8012d60 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:52:02.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.698+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8010310 0x7f1de8008320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:52:02.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.698+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:52:02.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:02.698+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:52:03.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:03.502+0000 7f1e0a240640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8010310 msgr2=0x7f1de8008320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:52:03.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:03.502+0000 7f1e09a3f640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8012d60 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:52:03.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:03.502+0000 7f1e0a240640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8010310 0x7f1de8008320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:52:03.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:03.502+0000 7f1e0923e640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:52:03.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:03.502+0000 7f1e0923e640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:52:05.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:05.090+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:52:05.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:05.090+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8006b80 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:52:05.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:05.090+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8010310 msgr2=0x7f1de8008320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:52:05.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:05.090+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8010310 0x7f1de8008320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:52:05.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:05.090+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8012d60 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:52:05.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:05.090+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8013830 0x7f1de8011d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:52:05.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:05.090+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415d710 con 0x7f1de8008860 2023-12-04T05:52:05.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:05.090+0000 7f1dfb7fe640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8013830 2023-12-04T05:52:05.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:05.090+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8008860 legacy=0x7f1de8012270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:52:05.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:05.090+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8013830 msgr2=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:52:05.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:05.090+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8013830 0x7f1de8011d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:52:05.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:05.294+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8008860 legacy=0x7f1de8012270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:52:05.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:05.294+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8013830 msgr2=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:52:05.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:05.294+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8013830 0x7f1de8011d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:52:05.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:05.698+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8008860 legacy=0x7f1de8012270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:52:05.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:05.698+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8013830 msgr2=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:52:05.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:05.698+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8013830 0x7f1de8011d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:52:06.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:06.502+0000 7f1e09a3f640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8013830 msgr2=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:52:06.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:06.502+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8008860 legacy=0x7f1de8012270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:52:06.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:06.502+0000 7f1e09a3f640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8013830 0x7f1de8011d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:52:08.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.090+0000 7f1dfb7fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8013830 msgr2=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:52:08.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.090+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8013830 0x7f1de8011d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:52:08.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.090+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8008860 legacy=0x7f1de8012270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:52:08.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.090+0000 7f1e0a240640 1 -- reap_dead start 2023-12-04T05:52:08.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.090+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8006b80 0x7f1de8009090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:52:08.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.090+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de8008320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:52:08.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.090+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415ad50 con 0x7f1de8013830 2023-12-04T05:52:08.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.090+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8006b80 2023-12-04T05:52:08.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.090+0000 7f1dfb7fe640 1 -- --> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80136f0 con 0x7f1de8010310 2023-12-04T05:52:08.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.090+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8006b80 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:52:08.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.090+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8013830 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:52:08.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.090+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8006b80 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:52:08.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.090+0000 7f1e0a240640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 msgr2=0x7f1de8008320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:52:08.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.090+0000 7f1e0a240640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de8008320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:52:08.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.294+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8006b80 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:52:08.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.294+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8013830 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:52:08.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.294+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8006b80 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:52:08.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.294+0000 7f1e0a240640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 msgr2=0x7f1de8008320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:52:08.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.294+0000 7f1e0a240640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de8008320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:52:08.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.698+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8006b80 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:52:08.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.698+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8013830 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:52:08.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.698+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8006b80 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:52:08.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.698+0000 7f1e0a240640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 msgr2=0x7f1de8008320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:52:08.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:08.698+0000 7f1e0a240640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de8008320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:52:09.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:09.502+0000 7f1e09a3f640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8006b80 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.107:3300/0 2023-12-04T05:52:09.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:09.502+0000 7f1e0923e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8013830 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2023-12-04T05:52:09.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:09.502+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8006b80 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:52:09.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:09.502+0000 7f1e0a240640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 msgr2=0x7f1de8008320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3301/0 2023-12-04T05:52:09.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:09.502+0000 7f1e0a240640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de8008320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:52:11.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:11.090+0000 7f1dfb7fe640 1 -- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 msgr2=0x7f1de8008320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:52:11.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:11.090+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.195:3301/0,v1:172.21.15.195:6790/0] conn(0x7f1de8010310 0x7f1de8008320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:52:11.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:11.090+0000 7f1dfb7fe640 1 -- >> v2:172.21.15.107:3300/0 conn(0x7f1de8006b80 msgr2=0x7f1de8009090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:52:11.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:11.090+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1de8006b80 0x7f1de8009090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T05:52:11.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:11.090+0000 7f1dfb7fe640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7f1de8013830 legacy=0x7f1de8007c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T05:52:11.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:11.090+0000 7f1dfb7fe640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8008860 0x7f1de8011d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T05:52:11.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:11.090+0000 7f1dfb7fe640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0415d710 con 0x7f1de8012d60 2023-12-04T05:52:11.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:11.090+0000 7f1dfb7fe640 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de800aef0 con 0x7f1de8008860 2023-12-04T05:52:11.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:11.090+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8008860 msgr2=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:52:11.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:11.090+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8012d60 legacy=0x7f1de8012270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:52:11.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:11.090+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8008860 0x7f1de8011d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T05:52:11.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:11.294+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8012d60 legacy=0x7f1de8012270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:52:11.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:11.294+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8008860 msgr2=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:52:11.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:11.294+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8008860 0x7f1de8011d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T05:52:11.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:11.698+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8012d60 legacy=0x7f1de8012270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:52:11.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:11.698+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8008860 msgr2=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:52:11.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:11.698+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8008860 0x7f1de8011d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T05:52:12.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:12.502+0000 7f1e09a3f640 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f1de8008860 msgr2=0x7f1de8011d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2023-12-04T05:52:12.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:12.502+0000 7f1e0923e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f1de8012d60 legacy=0x7f1de8012270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2023-12-04T05:52:12.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-04T05:52:12.502+0000 7f1e09a3f640 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f1de8008860 0x7f1de8011d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T05:52:13.972 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-04T05:52:13.973 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-12-04T05:52:13.985 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-04T05:52:13.985 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-04T05:52:13.985 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T05:52:13.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T05:52:13.988 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a' 2023-12-04T05:52:14.005 INFO:tasks.ceph.mds.a:Stopped 2023-12-04T05:52:14.005 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-04T05:52:14.005 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T05:52:14.006 INFO:tasks.ceph.mds.c:Stopped 2023-12-04T05:52:14.006 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-12-04T05:52:14.006 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T05:52:14.006 INFO:tasks.ceph.mds.e:Stopped 2023-12-04T05:52:14.007 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-04T05:52:14.007 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T05:52:14.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T05:52:14.008 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi195 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b' 2023-12-04T05:52:14.021 INFO:tasks.ceph.mds.b:Stopped 2023-12-04T05:52:14.021 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-04T05:52:14.021 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T05:52:14.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T05:52:14.022 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi195 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' 2023-12-04T05:52:14.039 INFO:tasks.ceph.mds.d:Stopped 2023-12-04T05:52:14.039 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-12-04T05:52:14.040 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T05:52:14.040 INFO:tasks.ceph.mds.f:Stopped 2023-12-04T05:52:14.040 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-04T05:52:14.041 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-04T05:52:14.041 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T05:52:14.041 INFO:tasks.ceph.osd.0:Stopped 2023-12-04T05:52:14.042 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-04T05:52:14.042 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T05:52:14.042 INFO:tasks.ceph.osd.1:Stopped 2023-12-04T05:52:14.042 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-04T05:52:14.043 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T05:52:14.044 INFO:tasks.ceph.osd.2:Stopped 2023-12-04T05:52:14.045 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-12-04T05:52:14.045 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T05:52:14.045 INFO:tasks.ceph.osd.3:Stopped 2023-12-04T05:52:14.045 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-12-04T05:52:14.045 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T05:52:14.045 INFO:tasks.ceph.osd.4:Stopped 2023-12-04T05:52:14.046 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-12-04T05:52:14.046 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T05:52:14.046 INFO:tasks.ceph.osd.5:Stopped 2023-12-04T05:52:14.046 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-12-04T05:52:14.046 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T05:52:14.047 INFO:tasks.ceph.osd.6:Stopped 2023-12-04T05:52:14.047 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-12-04T05:52:14.047 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T05:52:14.047 INFO:tasks.ceph.osd.7:Stopped 2023-12-04T05:52:14.047 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-04T05:52:14.047 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-12-04T05:52:14.047 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T05:52:14.048 INFO:tasks.ceph.mgr.y:Stopped 2023-12-04T05:52:14.048 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-04T05:52:14.048 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T05:52:14.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T05:52:14.048 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi195 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' 2023-12-04T05:52:14.060 INFO:tasks.ceph.mgr.x:Stopped 2023-12-04T05:52:14.060 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-04T05:52:14.060 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-04T05:52:14.060 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T05:52:14.063 INFO:tasks.ceph.mon.a:Stopped 2023-12-04T05:52:14.064 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-04T05:52:14.064 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T05:52:14.065 INFO:tasks.ceph.mon.b:Stopped 2023-12-04T05:52:14.065 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-04T05:52:14.065 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T05:52:14.065 INFO:tasks.ceph.mon.c:Stopped 2023-12-04T05:52:14.066 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-04T05:52:14.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v 'Replacing daemon mds' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2023-12-04T05:52:14.105 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi107.front.sepia.ceph.com 2023-12-04T05:52:14.105 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-04T05:52:14.349 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi107.front.sepia.ceph.com 2023-12-04T05:52:14.349 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-04T05:52:14.536 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi107.front.sepia.ceph.com 2023-12-04T05:52:14.536 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-04T05:52:14.702 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi107.front.sepia.ceph.com 2023-12-04T05:52:14.702 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-12-04T05:52:14.885 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi195.front.sepia.ceph.com 2023-12-04T05:52:14.886 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-12-04T05:52:15.036 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi195.front.sepia.ceph.com 2023-12-04T05:52:15.036 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-12-04T05:52:15.193 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi195.front.sepia.ceph.com 2023-12-04T05:52:15.194 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-12-04T05:52:15.352 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi195.front.sepia.ceph.com 2023-12-04T05:52:15.353 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-12-04T05:52:15.512 INFO:tasks.ceph:Archiving mon data... 2023-12-04T05:52:15.518 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/vshankar-2023-12-04_05:06:51-fs-wip-vshankar-testing-20231127.102654-distro-default-smithi/7476686/data/mon.a.tgz 2023-12-04T05:52:15.518 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2023-12-04T05:52:15.526 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.aAQhUu4BJh 2023-12-04T05:52:15.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.aAQhUu4BJh 2023-12-04T05:52:15.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.aAQhUu4BJh 2023-12-04T05:52:15.709 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.aAQhUu4BJh is 259KB 2023-12-04T05:52:15.771 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.aAQhUu4BJh 2023-12-04T05:52:15.778 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/vshankar-2023-12-04_05:06:51-fs-wip-vshankar-testing-20231127.102654-distro-default-smithi/7476686/data/mon.b.tgz 2023-12-04T05:52:15.779 DEBUG:teuthology.orchestra.run.smithi195:> mktemp 2023-12-04T05:52:15.785 INFO:teuthology.orchestra.run.smithi195.stdout:/tmp/tmp.ykS31pTzdD 2023-12-04T05:52:15.786 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.ykS31pTzdD 2023-12-04T05:52:15.902 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0666 /tmp/tmp.ykS31pTzdD 2023-12-04T05:52:15.969 DEBUG:teuthology.orchestra.remote:smithi195:/tmp/tmp.ykS31pTzdD is 261KB 2023-12-04T05:52:16.027 DEBUG:teuthology.orchestra.run.smithi195:> rm -fr /tmp/tmp.ykS31pTzdD 2023-12-04T05:52:16.033 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/vshankar-2023-12-04_05:06:51-fs-wip-vshankar-testing-20231127.102654-distro-default-smithi/7476686/data/mon.c.tgz 2023-12-04T05:52:16.033 DEBUG:teuthology.orchestra.run.smithi195:> mktemp 2023-12-04T05:52:16.081 INFO:teuthology.orchestra.run.smithi195.stdout:/tmp/tmp.rjc0zDQlod 2023-12-04T05:52:16.081 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.rjc0zDQlod 2023-12-04T05:52:16.198 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0666 /tmp/tmp.rjc0zDQlod 2023-12-04T05:52:16.260 DEBUG:teuthology.orchestra.remote:smithi195:/tmp/tmp.rjc0zDQlod is 261KB 2023-12-04T05:52:16.326 DEBUG:teuthology.orchestra.run.smithi195:> rm -fr /tmp/tmp.rjc0zDQlod 2023-12-04T05:52:16.331 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-04T05:52:16.331 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-04T05:52:16.334 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-04T05:52:16.382 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-12-04T05:52:16.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-12-04T05:52:16.392 DEBUG:teuthology.orchestra.run.smithi195:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-12-04T05:52:16.418 INFO:teuthology.orchestra.run.smithi107.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-12-04T05:52:16.449 INFO:teuthology.orchestra.run.smithi195.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-12-04T05:52:16.454 INFO:tasks.ceph:Archiving crash dumps... 2023-12-04T05:52:16.456 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2023-12-04_05:06:51-fs-wip-vshankar-testing-20231127.102654-distro-default-smithi/7476686/remote/smithi107/crash 2023-12-04T05:52:16.456 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-04T05:52:16.517 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2023-12-04_05:06:51-fs-wip-vshankar-testing-20231127.102654-distro-default-smithi/7476686/remote/smithi195/crash 2023-12-04T05:52:16.518 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-04T05:52:16.609 INFO:tasks.ceph:Shutting down logrotate 2023-12-04T05:52:16.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:52:16.613 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T05:52:16.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-12-04T05:52:16.714 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-12-04T05:52:16.737 INFO:tasks.ceph:Compressing logs... 2023-12-04T05:52:16.737 DEBUG:teuthology.orchestra.run.smithi107:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-04T05:52:16.759 DEBUG:teuthology.orchestra.run.smithi195:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-04T05:52:16.780 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52284.log 2023-12-04T05:52:16.781 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46229.log 2023-12-04T05:52:16.781 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.52284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50254.log 2023-12-04T05:52:16.784 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.46229.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52284.log.gz 2023-12-04T05:52:16.784 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47835.log 2023-12-04T05:52:16.785 INFO:teuthology.orchestra.run.smithi107.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46229.log.gz 2023-12-04T05:52:16.785 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47017.log 2023-12-04T05:52:16.785 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.47835.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50254.log.gz 2023-12-04T05:52:16.785 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50288.log 2023-12-04T05:52:16.785 INFO:teuthology.orchestra.run.smithi107.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47835.log.gz 2023-12-04T05:52:16.785 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.47017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51102.log 2023-12-04T05:52:16.785 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50288.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47017.log.gz 2023-12-04T05:52:16.785 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44580.log 2023-12-04T05:52:16.785 INFO:teuthology.orchestra.run.smithi107.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50288.log.gz 2023-12-04T05:52:16.785 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51102.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-12-04T05:52:16.786 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44580.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51102.log.gz 2023-12-04T05:52:16.786 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44528.log 2023-12-04T05:52:16.786 INFO:teuthology.orchestra.run.smithi107.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44580.log.gz 2023-12-04T05:52:16.786 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48511.log 2023-12-04T05:52:16.791 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46281.log 2023-12-04T05:52:16.791 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.48511.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44528.log.gz 2023-12-04T05:52:16.791 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46307.log 2023-12-04T05:52:16.791 INFO:teuthology.orchestra.run.smithi107.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48511.log.gz 2023-12-04T05:52:16.791 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.46281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45431.log 2023-12-04T05:52:16.791 INFO:teuthology.orchestra.run.smithi107.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46281.log.gz 2023-12-04T05:52:16.791 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.46307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46099.log 2023-12-04T05:52:16.791 INFO:teuthology.orchestra.run.smithi107.stderr: 87.8%/var/log/ceph/ceph-client.admin.45431.log: -- replaced with /var/log/ceph/ceph-client.admin.46307.log.gz 2023-12-04T05:52:16.791 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50817.log 2023-12-04T05:52:16.792 INFO:teuthology.orchestra.run.smithi107.stderr: 88.6%/var/log/ceph/ceph-client.admin.46099.log: -- replaced with /var/log/ceph/ceph-client.admin.45431.log.gz 2023-12-04T05:52:16.792 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50475.log 2023-12-04T05:52:16.792 INFO:teuthology.orchestra.run.smithi107.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46099.log.gz 2023-12-04T05:52:16.792 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50075.log 2023-12-04T05:52:16.792 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50475.log: gzip 88.5% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.50817.log.gz -- 2023-12-04T05:52:16.792 INFO:teuthology.orchestra.run.smithi107.stderr: /var/log/ceph/ceph-client.admin.45818.log 2023-12-04T05:52:16.792 INFO:teuthology.orchestra.run.smithi107.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50475.log.gz 2023-12-04T05:52:16.792 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45614.log 2023-12-04T05:52:16.792 INFO:teuthology.orchestra.run.smithi107.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50075.log.gz 2023-12-04T05:52:16.793 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.45818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48899.log 2023-12-04T05:52:16.793 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.45614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50952.log 2023-12-04T05:52:16.793 INFO:teuthology.orchestra.run.smithi107.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45614.log.gz 2023-12-04T05:52:16.793 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.48899.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.45818.log.gz 2023-12-04T05:52:16.793 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-12-04T05:52:16.793 INFO:teuthology.orchestra.run.smithi107.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.48899.log.gz 2023-12-04T05:52:16.793 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51338.log 2023-12-04T05:52:16.793 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50952.log: /var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38567.log 2023-12-04T05:52:16.793 INFO:teuthology.orchestra.run.smithi107.stderr:gzip 89.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.50952.log.gz -- 2023-12-04T05:52:16.793 INFO:teuthology.orchestra.run.smithi107.stderr: /var/log/ceph/ceph-client.admin.50900.log 2023-12-04T05:52:16.794 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51338.log: /var/log/ceph/ceph-client.admin.38567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51286.log 2023-12-04T05:52:16.794 INFO:teuthology.orchestra.run.smithi107.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51338.log.gz 2023-12-04T05:52:16.794 INFO:teuthology.orchestra.run.smithi107.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38567.log.gz 2023-12-04T05:52:16.802 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46385.log 2023-12-04T05:52:16.803 INFO:teuthology.orchestra.run.smithi107.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50900.log.gz 2023-12-04T05:52:16.803 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50679.log 2023-12-04T05:52:16.803 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.46385.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51286.log.gz 2023-12-04T05:52:16.803 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51835.log 2023-12-04T05:52:16.803 INFO:teuthology.orchestra.run.smithi107.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46385.log.gz 2023-12-04T05:52:16.803 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50343.log 2023-12-04T05:52:16.803 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50679.log: /var/log/ceph/ceph-client.admin.51835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52572.log 2023-12-04T05:52:16.803 INFO:teuthology.orchestra.run.smithi107.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50679.log.gz 2023-12-04T05:52:16.803 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50343.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.51835.log.gz -5 2023-12-04T05:52:16.804 INFO:teuthology.orchestra.run.smithi107.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52152.log 2023-12-04T05:52:16.804 INFO:teuthology.orchestra.run.smithi107.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50343.log.gz 2023-12-04T05:52:16.804 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.52572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50023.log 2023-12-04T05:52:16.804 INFO:teuthology.orchestra.run.smithi107.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52572.log.gz 2023-12-04T05:52:16.804 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.52152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51206.log 2023-12-04T05:52:16.804 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50023.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52152.log.gz 2023-12-04T05:52:16.804 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44290.log 2023-12-04T05:52:16.804 INFO:teuthology.orchestra.run.smithi107.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50023.log.gz 2023-12-04T05:52:16.804 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46333.log 2023-12-04T05:52:16.804 INFO:teuthology.orchestra.run.smithi107.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51206.log.gz 2023-12-04T05:52:16.805 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52019.log 2023-12-04T05:52:16.805 INFO:teuthology.orchestra.run.smithi107.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44290.log.gz 2023-12-04T05:52:16.805 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51443.log 2023-12-04T05:52:16.805 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.46333.log: /var/log/ceph/ceph-client.admin.52019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48234.log 2023-12-04T05:52:16.805 INFO:teuthology.orchestra.run.smithi107.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46333.log.gz 2023-12-04T05:52:16.805 INFO:teuthology.orchestra.run.smithi107.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52019.log.gz 2023-12-04T05:52:16.805 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44184.log 2023-12-04T05:52:16.805 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52321.log 2023-12-04T05:52:16.805 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.48234.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51443.log.gz 2023-12-04T05:52:16.806 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46753.log 2023-12-04T05:52:16.806 INFO:teuthology.orchestra.run.smithi107.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48234.log.gz 2023-12-04T05:52:16.806 INFO:teuthology.orchestra.run.smithi107.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44184.log.gz 2023-12-04T05:52:16.806 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.52321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45561.log 2023-12-04T05:52:16.806 INFO:teuthology.orchestra.run.smithi107.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52321.log.gz 2023-12-04T05:52:16.806 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.46753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50996.log 2023-12-04T05:52:16.806 INFO:teuthology.orchestra.run.smithi107.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46753.log.gz 2023-12-04T05:52:16.806 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.45561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44915.log 2023-12-04T05:52:16.806 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50996.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45561.log.gz 2023-12-04T05:52:16.807 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-12-04T05:52:16.807 INFO:teuthology.orchestra.run.smithi107.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50996.log.gz 2023-12-04T05:52:16.807 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45211.log 2023-12-04T05:52:16.807 INFO:teuthology.orchestra.run.smithi107.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44915.log.gz 2023-12-04T05:52:16.807 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46125.log 2023-12-04T05:52:16.807 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.45211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44709.log 2023-12-04T05:52:16.807 INFO:teuthology.orchestra.run.smithi107.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45211.log.gz 2023-12-04T05:52:16.807 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.46125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49996.log 2023-12-04T05:52:16.807 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44709.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46125.log.gz 2023-12-04T05:52:16.808 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51390.log 2023-12-04T05:52:16.808 INFO:teuthology.orchestra.run.smithi107.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44709.log.gz 2023-12-04T05:52:16.808 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.49996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52230.log 2023-12-04T05:52:16.808 INFO:teuthology.orchestra.run.smithi107.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49996.log.gz 2023-12-04T05:52:16.808 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52496.log 2023-12-04T05:52:16.808 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51364.log 2023-12-04T05:52:16.808 INFO:teuthology.orchestra.run.smithi107.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51390.log.gz 2023-12-04T05:52:16.808 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.52230.log: /var/log/ceph/ceph-client.admin.52496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51809.log 2023-12-04T05:52:16.808 INFO:teuthology.orchestra.run.smithi107.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52230.log.gz 2023-12-04T05:52:16.818 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38385.log 2023-12-04T05:52:16.818 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-12-04T05:52:16.818 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.38385.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-12-04T05:52:16.818 INFO:teuthology.orchestra.run.smithi195.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38385.log.gz 2023-12-04T05:52:16.818 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38433.log 2023-12-04T05:52:16.819 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-12-04T05:52:16.819 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.38433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38457.log 2023-12-04T05:52:16.819 INFO:teuthology.orchestra.run.smithi195.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38433.log.gz 2023-12-04T05:52:16.819 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38505.log 2023-12-04T05:52:16.819 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.38457.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-12-04T05:52:16.819 INFO:teuthology.orchestra.run.smithi195.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38457.log.gz 2023-12-04T05:52:16.819 INFO:teuthology.orchestra.run.smithi195.stderr:gzip/var/log/ceph/ceph-client.admin.38505.log: -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2023-12-04T05:52:16.819 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-mgr.x.log: gzip 88.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.38505.log.gz --verbose 2023-12-04T05:52:16.819 INFO:teuthology.orchestra.run.smithi195.stderr: -- /var/log/ceph/ceph.log 2023-12-04T05:52:16.820 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-12-04T05:52:16.820 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38553.log 2023-12-04T05:52:16.820 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38529.log 2023-12-04T05:52:16.820 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.38553.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38553.log.gz 2023-12-04T05:52:16.820 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-04T05:52:16.820 INFO:teuthology.orchestra.run.smithi195.stderr: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-04T05:52:16.820 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.38529.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38529.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-12-04T05:52:16.820 INFO:teuthology.orchestra.run.smithi195.stderr: 2023-12-04T05:52:16.820 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-12-04T05:52:16.820 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-mds.d.log: 93.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-04T05:52:16.821 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-12-04T05:52:16.823 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-mds.b.log: 94.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-12-04T05:52:16.824 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-12-04T05:52:16.831 INFO:teuthology.orchestra.run.smithi107.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52496.log.gz 2023-12-04T05:52:16.832 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51364.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-12-04T05:52:16.832 INFO:teuthology.orchestra.run.smithi107.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51364.log.gz 2023-12-04T05:52:16.832 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44710.log 2023-12-04T05:52:16.832 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51809.log: /var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50184.log 2023-12-04T05:52:16.832 INFO:teuthology.orchestra.run.smithi107.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51809.log.gz 2023-12-04T05:52:16.832 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49061.log 2023-12-04T05:52:16.832 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46463.log 2023-12-04T05:52:16.832 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50184.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44710.log.gz 2023-12-04T05:52:16.833 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.49061.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50184.log.gz 2023-12-04T05:52:16.833 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50843.log 2023-12-04T05:52:16.833 INFO:teuthology.orchestra.run.smithi107.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49061.log.gz 2023-12-04T05:52:16.833 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52311.log 2023-12-04T05:52:16.833 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.46463.log: /var/log/ceph/ceph-client.admin.50843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51495.log 2023-12-04T05:52:16.833 INFO:teuthology.orchestra.run.smithi107.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46463.log.gz 2023-12-04T05:52:16.833 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52363.log 2023-12-04T05:52:16.833 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.52311.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.50843.log.gz 2023-12-04T05:52:16.833 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51495.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52311.log.gz 2023-12-04T05:52:16.834 INFO:teuthology.orchestra.run.smithi107.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51495.log.gz 2023-12-04T05:52:16.834 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51783.log 2023-12-04T05:52:16.834 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.52363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44104.log 2023-12-04T05:52:16.834 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44398.log 2023-12-04T05:52:16.834 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51783.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52363.log.gz 2023-12-04T05:52:16.834 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52670.log 2023-12-04T05:52:16.834 INFO:teuthology.orchestra.run.smithi107.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51783.log.gz 2023-12-04T05:52:16.834 INFO:teuthology.orchestra.run.smithi107.stderr:gzip/var/log/ceph/ceph-client.admin.44398.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51180.log 2023-12-04T05:52:16.834 INFO:teuthology.orchestra.run.smithi107.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44104.log.gz 2023-12-04T05:52:16.835 INFO:teuthology.orchestra.run.smithi107.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44398.log.gz 2023-12-04T05:52:16.835 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.52670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46151.log 2023-12-04T05:52:16.835 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49787.log 2023-12-04T05:52:16.835 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51180.log: /var/log/ceph/ceph-client.admin.46151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50626.log 2023-12-04T05:52:16.835 INFO:teuthology.orchestra.run.smithi107.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51180.log.gz 2023-12-04T05:52:16.835 INFO:teuthology.orchestra.run.smithi107.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46151.log.gz 2023-12-04T05:52:16.835 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.49787.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49787.log.gz 2023-12-04T05:52:16.835 INFO:teuthology.orchestra.run.smithi107.stderr: 95.6% -- replaced with /var/log/ceph/ceph-client.admin.52670.log.gz 2023-12-04T05:52:16.835 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51601.log 2023-12-04T05:52:16.835 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45588.log 2023-12-04T05:52:16.836 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51601.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50626.log.gz 2023-12-04T05:52:16.836 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47776.log 2023-12-04T05:52:16.836 INFO:teuthology.orchestra.run.smithi107.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51601.log.gz 2023-12-04T05:52:16.836 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.45588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44606.log 2023-12-04T05:52:16.836 INFO:teuthology.orchestra.run.smithi107.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45588.log.gz 2023-12-04T05:52:16.836 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.47776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50926.log 2023-12-04T05:52:16.836 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45509.log 2023-12-04T05:52:16.836 INFO:teuthology.orchestra.run.smithi107.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47776.log.gz 2023-12-04T05:52:16.836 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50926.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44606.log.gz 2023-12-04T05:52:16.837 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44711.log 2023-12-04T05:52:16.837 INFO:teuthology.orchestra.run.smithi107.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50926.log.gz 2023-12-04T05:52:16.837 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49446.log 2023-12-04T05:52:16.837 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.45509.log: /var/log/ceph/ceph-client.admin.44711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45325.log 2023-12-04T05:52:16.837 INFO:teuthology.orchestra.run.smithi107.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45509.log.gz/var/log/ceph/ceph-client.admin.49446.log: 2023-12-04T05:52:16.837 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51653.log 2023-12-04T05:52:16.837 INFO:teuthology.orchestra.run.smithi107.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44711.log.gz 2023-12-04T05:52:16.837 INFO:teuthology.orchestra.run.smithi107.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49446.log.gz 2023-12-04T05:52:16.837 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.45325.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45325.log.gz 2023-12-04T05:52:16.838 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47356.log 2023-12-04T05:52:16.838 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45768.log 2023-12-04T05:52:16.838 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.47356.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51653.log.gz 2023-12-04T05:52:16.838 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-12-04T05:52:16.838 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.45768.log: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52443.log 2023-12-04T05:52:16.838 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47356.log.gz 2023-12-04T05:52:16.838 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51234.log 2023-12-04T05:52:16.838 INFO:teuthology.orchestra.run.smithi107.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.45768.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45923.log 2023-12-04T05:52:16.838 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-12-04T05:52:16.839 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.52443.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52443.log.gz 2023-12-04T05:52:16.839 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51234.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51234.log.gz 2023-12-04T05:52:16.839 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43916.log 2023-12-04T05:52:16.839 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.45923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49970.log 2023-12-04T05:52:16.839 INFO:teuthology.orchestra.run.smithi107.stderr: 89.4%/var/log/ceph/ceph-client.admin.43916.log: -- replaced with /var/log/ceph/ceph-client.admin.45923.log.gz 2023-12-04T05:52:16.839 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46779.log 2023-12-04T05:52:16.839 INFO:teuthology.orchestra.run.smithi107.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43916.log.gz 2023-12-04T05:52:16.839 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.49970.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49970.log.gz 2023-12-04T05:52:16.839 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49245.log 2023-12-04T05:52:16.839 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.46779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44344.log 2023-12-04T05:52:16.840 INFO:teuthology.orchestra.run.smithi107.stderr: 88.8%/var/log/ceph/ceph-client.admin.49245.log: -- replaced with /var/log/ceph/ceph-client.admin.46779.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44658.log 2023-12-04T05:52:16.840 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-12-04T05:52:16.840 INFO:teuthology.orchestra.run.smithi107.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49245.log.gz 2023-12-04T05:52:16.840 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44344.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44344.log.gz 2023-12-04T05:52:16.840 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46489.log 2023-12-04T05:52:16.840 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44914.log 2023-12-04T05:52:16.840 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44658.log: /var/log/ceph/ceph-client.admin.46489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50210.log 2023-12-04T05:52:16.840 INFO:teuthology.orchestra.run.smithi107.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44658.log.gz 2023-12-04T05:52:16.840 INFO:teuthology.orchestra.run.smithi107.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46489.log.gz 2023-12-04T05:52:16.841 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44914.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44914.log.gz 2023-12-04T05:52:16.841 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44020.log 2023-12-04T05:52:16.841 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49363.log 2023-12-04T05:52:16.841 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44020.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50210.log.gz 2023-12-04T05:52:16.841 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-12-04T05:52:16.841 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44908.log 2023-12-04T05:52:16.841 INFO:teuthology.orchestra.run.smithi107.stderr: 88.4%/var/log/ceph/ceph-client.admin.49363.log: -- replaced with /var/log/ceph/ceph-client.admin.44020.log.gz 2023-12-04T05:52:16.841 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-osd.1.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49363.log.gz 2023-12-04T05:52:16.841 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44900.log 2023-12-04T05:52:16.841 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49175.log 2023-12-04T05:52:16.842 INFO:teuthology.orchestra.run.smithi107.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44908.log.gz/var/log/ceph/ceph-client.admin.44900.log: 2023-12-04T05:52:16.842 INFO:teuthology.orchestra.run.smithi107.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44900.log.gz 2023-12-04T05:52:16.842 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52631.log 2023-12-04T05:52:16.842 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50582.log 2023-12-04T05:52:16.843 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38409.log 2023-12-04T05:52:16.851 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38481.log 2023-12-04T05:52:16.851 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.38409.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38409.log.gz 2023-12-04T05:52:16.854 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.49175.log: /var/log/ceph/ceph-client.admin.52631.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49175.log.gz 2023-12-04T05:52:16.854 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46724.log 2023-12-04T05:52:16.854 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50582.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50582.log.gz 2023-12-04T05:52:16.854 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45377.log 2023-12-04T05:52:16.854 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.46724.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46724.log.gz 2023-12-04T05:52:16.855 INFO:teuthology.orchestra.run.smithi107.stderr: 95.6% -- replaced with /var/log/ceph/ceph-client.admin.52631.log.gz 2023-12-04T05:52:16.855 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50529.log 2023-12-04T05:52:16.855 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.45377.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45377.log.gz 2023-12-04T05:52:16.855 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51312.log 2023-12-04T05:52:16.855 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45351.log 2023-12-04T05:52:16.855 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50529.log: /var/log/ceph/ceph-client.admin.51312.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50529.log.gz 2023-12-04T05:52:16.855 INFO:teuthology.orchestra.run.smithi107.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51312.log.gz 2023-12-04T05:52:16.855 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44916.log 2023-12-04T05:52:16.855 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.45351.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45351.log.gz 2023-12-04T05:52:16.856 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46672.log 2023-12-04T05:52:16.856 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52204.log 2023-12-04T05:52:16.856 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44916.log: /var/log/ceph/ceph-client.admin.46672.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44916.log.gz 2023-12-04T05:52:16.856 INFO:teuthology.orchestra.run.smithi107.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46672.log.gz 2023-12-04T05:52:16.856 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51260.log 2023-12-04T05:52:16.856 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45535.log 2023-12-04T05:52:16.856 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.52204.log: /var/log/ceph/ceph-client.admin.51260.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52204.log.gz 2023-12-04T05:52:16.856 INFO:teuthology.orchestra.run.smithi107.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51260.log.gz 2023-12-04T05:52:16.856 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46411.log 2023-12-04T05:52:16.856 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48664.log 2023-12-04T05:52:16.857 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.45535.log: /var/log/ceph/ceph-client.admin.46411.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45535.log.gz 2023-12-04T05:52:16.857 INFO:teuthology.orchestra.run.smithi107.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46411.log.gzgzip 2023-12-04T05:52:16.857 INFO:teuthology.orchestra.run.smithi107.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44372.log 2023-12-04T05:52:16.857 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.48664.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48664.log.gz 2023-12-04T05:52:16.857 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50501.log 2023-12-04T05:52:16.857 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45271.log 2023-12-04T05:52:16.857 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44372.log: /var/log/ceph/ceph-client.admin.50501.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44372.log.gz 2023-12-04T05:52:16.857 INFO:teuthology.orchestra.run.smithi107.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50501.log.gz 2023-12-04T05:52:16.857 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51731.log 2023-12-04T05:52:16.858 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.45271.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45271.log.gz 2023-12-04T05:52:16.858 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45642.log 2023-12-04T05:52:16.858 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44424.log 2023-12-04T05:52:16.858 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51731.log: /var/log/ceph/ceph-client.admin.45642.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51731.log.gz 2023-12-04T05:52:16.858 INFO:teuthology.orchestra.run.smithi107.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45642.log.gz 2023-12-04T05:52:16.858 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50784.log 2023-12-04T05:52:16.858 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51627.log 2023-12-04T05:52:16.858 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44424.log: /var/log/ceph/ceph-client.admin.50784.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44424.log.gz 2023-12-04T05:52:16.858 INFO:teuthology.orchestra.run.smithi107.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50784.log.gz 2023-12-04T05:52:16.859 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-04T05:52:16.859 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50101.log 2023-12-04T05:52:16.859 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51627.log: /var/log/ceph/ceph.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51627.log.gz 2023-12-04T05:52:16.859 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46541.log 2023-12-04T05:52:16.859 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50101.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50101.log.gz 2023-12-04T05:52:16.859 INFO:teuthology.orchestra.run.smithi107.stderr: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-04T05:52:16.859 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51705.log 2023-12-04T05:52:16.859 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.46541.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46541.log.gz 2023-12-04T05:52:16.859 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52126.log 2023-12-04T05:52:16.859 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52600.log 2023-12-04T05:52:16.860 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51705.log: /var/log/ceph/ceph-client.admin.52126.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51705.log.gz 2023-12-04T05:52:16.860 INFO:teuthology.orchestra.run.smithi107.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52126.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44262.log 2023-12-04T05:52:16.860 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-12-04T05:52:16.863 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.52600.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52600.log.gz 2023-12-04T05:52:16.863 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51521.log 2023-12-04T05:52:16.864 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44318.log 2023-12-04T05:52:16.864 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44262.log: /var/log/ceph/ceph-client.admin.51521.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44262.log.gz 2023-12-04T05:52:16.864 INFO:teuthology.orchestra.run.smithi107.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51521.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46177.log 2023-12-04T05:52:16.864 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-12-04T05:52:16.864 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44318.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44318.log.gz 2023-12-04T05:52:16.864 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50758.log 2023-12-04T05:52:16.864 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51547.log 2023-12-04T05:52:16.864 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.46177.log: /var/log/ceph/ceph-client.admin.50758.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46177.log.gz 2023-12-04T05:52:16.864 INFO:teuthology.orchestra.run.smithi107.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50758.log.gz 2023-12-04T05:52:16.865 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51023.log 2023-12-04T05:52:16.865 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51547.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51547.log.gz 2023-12-04T05:52:16.865 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52098.log 2023-12-04T05:52:16.865 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46359.log 2023-12-04T05:52:16.865 INFO:teuthology.orchestra.run.smithi107.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51023.log.gz 2023-12-04T05:52:16.865 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.52098.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52098.log.gz 2023-12-04T05:52:16.865 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44918.log 2023-12-04T05:52:16.865 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44708.log 2023-12-04T05:52:16.866 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.38481.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38481.log.gz 2023-12-04T05:52:16.868 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.46359.log: /var/log/ceph/ceph-client.admin.44918.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46359.log.gz 2023-12-04T05:52:16.869 INFO:teuthology.orchestra.run.smithi107.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44918.log.gz 2023-12-04T05:52:16.869 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44702.log 2023-12-04T05:52:16.869 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44708.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44708.log.gz 2023-12-04T05:52:16.869 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52546.log 2023-12-04T05:52:16.869 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52391.log 2023-12-04T05:52:16.869 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44702.log: /var/log/ceph/ceph-client.admin.52546.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44702.log.gz 2023-12-04T05:52:16.869 INFO:teuthology.orchestra.run.smithi107.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52546.log.gz 2023-12-04T05:52:16.869 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45185.log 2023-12-04T05:52:16.870 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51757.log 2023-12-04T05:52:16.870 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.52391.log: /var/log/ceph/ceph-client.admin.45185.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52391.log.gz 2023-12-04T05:52:16.870 INFO:teuthology.orchestra.run.smithi107.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45185.log.gz 2023-12-04T05:52:16.870 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44158.log 2023-12-04T05:52:16.870 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51154.log 2023-12-04T05:52:16.872 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44158.log: /var/log/ceph/ceph-client.admin.51757.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51757.log.gz 2023-12-04T05:52:16.872 INFO:teuthology.orchestra.run.smithi107.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44158.log.gz 2023-12-04T05:52:16.872 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46515.log 2023-12-04T05:52:16.872 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38447.log 2023-12-04T05:52:16.872 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51154.log: /var/log/ceph/ceph-client.admin.46515.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51154.log.gz 2023-12-04T05:52:16.873 INFO:teuthology.orchestra.run.smithi107.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46515.log.gz 2023-12-04T05:52:16.873 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44210.log 2023-12-04T05:52:16.873 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51861.log 2023-12-04T05:52:16.873 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.38447.log: /var/log/ceph/ceph-client.admin.44210.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38447.log.gz 2023-12-04T05:52:16.873 INFO:teuthology.orchestra.run.smithi107.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44210.log.gz 2023-12-04T05:52:16.873 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45718.log 2023-12-04T05:52:16.874 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45245.log 2023-12-04T05:52:16.874 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51861.log: /var/log/ceph/ceph-client.admin.45718.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51861.log.gz 2023-12-04T05:52:16.875 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45100.log 2023-12-04T05:52:16.875 INFO:teuthology.orchestra.run.smithi107.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.45718.log.gz 2023-12-04T05:52:16.875 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.45245.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45245.log.gz 2023-12-04T05:52:16.875 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50314.log 2023-12-04T05:52:16.876 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45405.log 2023-12-04T05:52:16.876 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.45100.log: /var/log/ceph/ceph-client.admin.50314.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45100.log.gz 2023-12-04T05:52:16.876 INFO:teuthology.orchestra.run.smithi107.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50314.log.gz 2023-12-04T05:52:16.876 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51679.log 2023-12-04T05:52:16.877 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51993.log 2023-12-04T05:52:16.877 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.45405.log: /var/log/ceph/ceph-client.admin.51679.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45405.log.gz 2023-12-04T05:52:16.877 INFO:teuthology.orchestra.run.smithi107.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51679.log.gz 2023-12-04T05:52:16.878 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51575.log 2023-12-04T05:52:16.879 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44450.log 2023-12-04T05:52:16.879 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51993.log: /var/log/ceph/ceph-client.admin.51575.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51993.log.gz 2023-12-04T05:52:16.879 INFO:teuthology.orchestra.run.smithi107.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51575.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50555.log 2023-12-04T05:52:16.879 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-12-04T05:52:16.880 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44450.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44450.log.gz 2023-12-04T05:52:16.880 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50395.log 2023-12-04T05:52:16.880 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50158.log 2023-12-04T05:52:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50555.log: /var/log/ceph/ceph-client.admin.50395.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50555.log.gz 2023-12-04T05:52:16.881 INFO:teuthology.orchestra.run.smithi107.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50395.log.gz 2023-12-04T05:52:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-04T05:52:16.882 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49944.log 2023-12-04T05:52:16.883 INFO:teuthology.orchestra.run.smithi195.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-12-04T05:52:16.883 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50158.log: /var/log/ceph/ceph.audit.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50158.log.gz 2023-12-04T05:52:16.884 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52470.log 2023-12-04T05:52:16.884 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.49944.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49944.log.gz 2023-12-04T05:52:16.884 INFO:teuthology.orchestra.run.smithi107.stderr: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-04T05:52:16.884 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-12-04T05:52:16.885 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49892.log 2023-12-04T05:52:16.885 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.52470.log: /var/log/ceph/ceph-mds.c.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52470.log.gz 2023-12-04T05:52:16.885 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51887.log 2023-12-04T05:52:16.885 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.49892.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49892.log.gz 2023-12-04T05:52:16.887 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-12-04T05:52:16.887 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51887.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51887.log.gz 2023-12-04T05:52:16.888 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46437.log 2023-12-04T05:52:16.888 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-mds.a.log: 91.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-12-04T05:52:16.907 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52178.log 2023-12-04T05:52:16.907 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.46437.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46437.log.gz 2023-12-04T05:52:16.907 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50447.log 2023-12-04T05:52:16.908 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.52178.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52178.log.gz 2023-12-04T05:52:16.908 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51967.log 2023-12-04T05:52:16.909 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50447.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50447.log.gz 2023-12-04T05:52:16.910 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50127.log 2023-12-04T05:52:16.910 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51967.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51967.log.gz 2023-12-04T05:52:16.911 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44701.log 2023-12-04T05:52:16.912 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50127.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50127.log.gz 2023-12-04T05:52:16.912 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44919.log 2023-12-04T05:52:16.913 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44701.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44701.log.gz 2023-12-04T05:52:16.914 INFO:teuthology.orchestra.run.smithi195.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-12-04T05:52:16.914 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47999.log 2023-12-04T05:52:16.915 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44919.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44919.log.gz 2023-12-04T05:52:16.915 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44554.log 2023-12-04T05:52:16.916 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.47999.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47999.log.gz 2023-12-04T05:52:16.916 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49918.log 2023-12-04T05:52:16.917 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44554.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44554.log.gz 2023-12-04T05:52:16.917 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50369.log 2023-12-04T05:52:16.918 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.49918.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49918.log.gz 2023-12-04T05:52:16.918 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52045.log 2023-12-04T05:52:16.919 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50369.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50369.log.gz 2023-12-04T05:52:16.920 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45668.log 2023-12-04T05:52:16.921 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.52045.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52045.log.gz 2023-12-04T05:52:16.921 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51076.log 2023-12-04T05:52:16.922 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.45668.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.45668.log.gz 2023-12-04T05:52:16.923 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46698.log 2023-12-04T05:52:16.923 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51076.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51076.log.gz 2023-12-04T05:52:16.924 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44476.log 2023-12-04T05:52:16.925 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.46698.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46698.log.gz 2023-12-04T05:52:16.925 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51050.log 2023-12-04T05:52:16.927 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44476.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44476.log.gz 2023-12-04T05:52:16.927 INFO:teuthology.orchestra.run.smithi107.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-12-04T05:52:16.927 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49821.log 2023-12-04T05:52:16.927 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51050.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51050.log.gz 2023-12-04T05:52:16.928 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44236.log 2023-12-04T05:52:16.928 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44502.log 2023-12-04T05:52:16.929 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.49821.log: /var/log/ceph/ceph-client.admin.44236.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.49821.log.gz 2023-12-04T05:52:16.929 INFO:teuthology.orchestra.run.smithi107.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44236.log.gz 2023-12-04T05:52:16.929 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50872.log 2023-12-04T05:52:16.930 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44502.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44502.log.gz 2023-12-04T05:52:16.930 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51941.log 2023-12-04T05:52:16.931 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44706.log 2023-12-04T05:52:16.931 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50872.log: /var/log/ceph/ceph-client.admin.51941.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50872.log.gz 2023-12-04T05:52:16.932 INFO:teuthology.orchestra.run.smithi107.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51941.log.gz 2023-12-04T05:52:16.932 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51416.log 2023-12-04T05:52:16.932 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44706.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44706.log.gz 2023-12-04T05:52:16.933 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47222.log 2023-12-04T05:52:16.934 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45152.log 2023-12-04T05:52:16.934 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51416.log: /var/log/ceph/ceph-client.admin.47222.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51416.log.gz 2023-12-04T05:52:16.934 INFO:teuthology.orchestra.run.smithi107.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47222.log.gz 2023-12-04T05:52:16.934 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38331.log 2023-12-04T05:52:16.941 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44920.log 2023-12-04T05:52:16.941 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.45152.log: /var/log/ceph/ceph-client.admin.38331.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45152.log.gz 2023-12-04T05:52:16.941 INFO:teuthology.orchestra.run.smithi107.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.38331.log.gz 2023-12-04T05:52:16.941 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50705.log 2023-12-04T05:52:16.941 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44920.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44920.log.gz 2023-12-04T05:52:16.941 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44130.log 2023-12-04T05:52:16.941 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44712.log 2023-12-04T05:52:16.942 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50705.log: /var/log/ceph/ceph-client.admin.44130.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50705.log.gz 2023-12-04T05:52:16.942 INFO:teuthology.orchestra.run.smithi107.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44130.log.gz 2023-12-04T05:52:16.942 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45483.log 2023-12-04T05:52:16.942 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45299.log 2023-12-04T05:52:16.942 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44712.log: /var/log/ceph/ceph-client.admin.45483.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44712.log.gz 2023-12-04T05:52:16.942 INFO:teuthology.orchestra.run.smithi107.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45483.log.gz 2023-12-04T05:52:16.942 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52258.log 2023-12-04T05:52:16.942 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51915.log 2023-12-04T05:52:16.943 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.45299.log: /var/log/ceph/ceph-client.admin.52258.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.45299.log.gz 2023-12-04T05:52:16.943 INFO:teuthology.orchestra.run.smithi107.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52258.log.gz 2023-12-04T05:52:16.943 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46203.log 2023-12-04T05:52:16.943 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43984.log 2023-12-04T05:52:16.946 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51915.log: /var/log/ceph/ceph-client.admin.46203.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51915.log.gz 2023-12-04T05:52:16.946 INFO:teuthology.orchestra.run.smithi107.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46203.log.gz 2023-12-04T05:52:16.946 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50421.log 2023-12-04T05:52:16.946 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48081.log 2023-12-04T05:52:16.946 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.43984.log: /var/log/ceph/ceph-client.admin.50421.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43984.log.gz 2023-12-04T05:52:16.946 INFO:teuthology.orchestra.run.smithi107.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50421.log.gz 2023-12-04T05:52:16.946 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52522.log 2023-12-04T05:52:16.946 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44632.log 2023-12-04T05:52:16.946 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.48081.log: /var/log/ceph/ceph-client.admin.52522.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48081.log.gz 2023-12-04T05:52:16.947 INFO:teuthology.orchestra.run.smithi107.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52522.log.gz 2023-12-04T05:52:16.947 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2023-12-04T05:52:16.947 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51128.log 2023-12-04T05:52:16.947 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.44632.log: /var/log/ceph/ceph-mds.e.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44632.log.gz 2023-12-04T05:52:16.947 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50049.log 2023-12-04T05:52:16.949 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51128.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51128.log.gz 2023-12-04T05:52:16.949 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.33480.log 2023-12-04T05:52:16.949 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50049.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50049.log.gz 2023-12-04T05:52:16.949 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47469.log 2023-12-04T05:52:16.950 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph.tmp-client.admin.33480.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.33480.log.gz 2023-12-04T05:52:16.950 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51469.log 2023-12-04T05:52:16.951 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.47469.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47469.log.gz 2023-12-04T05:52:16.951 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46046.log 2023-12-04T05:52:16.951 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.51469.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51469.log.gz 2023-12-04T05:52:16.952 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45868.log 2023-12-04T05:52:16.961 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.0.46046.log: 91.1% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-12-04T05:52:16.961 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52071.log 2023-12-04T05:52:16.962 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.45868.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.45868.log.gz 2023-12-04T05:52:16.963 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50653.log 2023-12-04T05:52:16.964 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.52071.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52071.log.gz 2023-12-04T05:52:16.964 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50732.log 2023-12-04T05:52:16.965 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50653.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50653.log.gz 2023-12-04T05:52:16.965 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52417.log 2023-12-04T05:52:16.966 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.50732.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50732.log.gz 2023-12-04T05:52:16.966 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45126.log 2023-12-04T05:52:16.967 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.52417.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52417.log.gz 2023-12-04T05:52:16.968 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45457.log 2023-12-04T05:52:16.968 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.45126.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45126.log.gz 2023-12-04T05:52:16.969 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46255.log 2023-12-04T05:52:16.970 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.45457.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45457.log.gz 2023-12-04T05:52:16.971 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.46255.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46255.log.gz 2023-12-04T05:52:16.998 INFO:teuthology.orchestra.run.smithi107.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2023-12-04T05:52:17.041 INFO:teuthology.orchestra.run.smithi195.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-12-04T05:52:17.046 INFO:teuthology.orchestra.run.smithi195.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-12-04T05:52:17.104 INFO:teuthology.orchestra.run.smithi195.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-12-04T05:52:17.129 INFO:teuthology.orchestra.run.smithi107.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-12-04T05:52:17.166 INFO:teuthology.orchestra.run.smithi195.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-12-04T05:52:17.166 INFO:teuthology.orchestra.run.smithi107.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-12-04T05:52:17.172 INFO:teuthology.orchestra.run.smithi107.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-12-04T05:52:17.212 INFO:teuthology.orchestra.run.smithi107.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-12-04T05:52:17.243 INFO:teuthology.orchestra.run.smithi195.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-12-04T05:52:17.269 INFO:teuthology.orchestra.run.smithi195.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-12-04T05:52:17.281 INFO:teuthology.orchestra.run.smithi107.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-12-04T05:52:18.276 INFO:teuthology.orchestra.run.smithi107.stderr: 93.7% -- replaced with /var/log/ceph/ceph-client.0.46046.log.gz 2023-12-04T05:52:18.277 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-12-04T05:52:18.277 INFO:teuthology.orchestra.run.smithi107.stderr:real 0m1.517s 2023-12-04T05:52:18.277 INFO:teuthology.orchestra.run.smithi107.stderr:user 0m3.630s 2023-12-04T05:52:18.277 INFO:teuthology.orchestra.run.smithi107.stderr:sys 0m0.189s 2023-12-04T05:52:19.694 INFO:teuthology.orchestra.run.smithi195.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2023-12-04T05:52:19.696 INFO:teuthology.orchestra.run.smithi195.stderr: 2023-12-04T05:52:19.696 INFO:teuthology.orchestra.run.smithi195.stderr:real 0m2.912s 2023-12-04T05:52:19.696 INFO:teuthology.orchestra.run.smithi195.stderr:user 0m4.871s 2023-12-04T05:52:19.696 INFO:teuthology.orchestra.run.smithi195.stderr:sys 0m0.173s 2023-12-04T05:52:19.697 INFO:tasks.ceph:Archiving logs... 2023-12-04T05:52:19.697 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/log/ceph to /home/teuthworker/archive/vshankar-2023-12-04_05:06:51-fs-wip-vshankar-testing-20231127.102654-distro-default-smithi/7476686/remote/smithi107/log 2023-12-04T05:52:19.699 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-04T05:52:20.296 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/log/ceph to /home/teuthworker/archive/vshankar-2023-12-04_05:06:51-fs-wip-vshankar-testing-20231127.102654-distro-default-smithi/7476686/remote/smithi195/log 2023-12-04T05:52:20.297 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-04T05:52:20.864 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-12-04T05:52:20.882 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-04T05:52:20.898 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-12-04T05:52:20.907 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-04T05:52:20.907 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-04T05:52:20.915 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-04T05:52:20.975 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-04T05:52:20.984 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-04T05:52:20.985 DEBUG:teuthology.orchestra.run.smithi107:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-04T05:52:20.988 DEBUG:teuthology.orchestra.run.smithi195:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-04T05:52:20.999 INFO:teuthology.orchestra.run.smithi107.stdout: remote refid st t when poll reach delay offset jitter 2023-12-04T05:52:21.000 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================== 2023-12-04T05:52:21.000 INFO:teuthology.orchestra.run.smithi107.stdout:*hv01.front.sepi 44.4.53.2 2 u 37 64 377 0.088 -4.164 1.661 2023-12-04T05:52:21.000 INFO:teuthology.orchestra.run.smithi107.stdout:+hv02.front.sepi 63.231.80.2 3 u 44 64 377 0.078 -0.994 1.435 2023-12-04T05:52:21.000 INFO:teuthology.orchestra.run.smithi107.stdout:+hv03.front.sepi 74.6.168.72 3 u 38 64 377 0.093 -1.982 1.068 2023-12-04T05:52:21.000 INFO:teuthology.orchestra.run.smithi107.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 +0.000 0.000 2023-12-04T05:52:21.001 INFO:teuthology.orchestra.run.smithi195.stdout: remote refid st t when poll reach delay offset jitter 2023-12-04T05:52:21.001 INFO:teuthology.orchestra.run.smithi195.stdout:============================================================================== 2023-12-04T05:52:21.001 INFO:teuthology.orchestra.run.smithi195.stdout:*hv01.front.sepi 44.4.53.2 2 u 44 64 377 0.073 -2.179 0.933 2023-12-04T05:52:21.002 INFO:teuthology.orchestra.run.smithi195.stdout:+hv02.front.sepi 63.231.80.2 3 u 43 64 377 0.098 -2.643 1.053 2023-12-04T05:52:21.002 INFO:teuthology.orchestra.run.smithi195.stdout:+hv03.front.sepi 74.6.168.72 3 u 42 64 377 0.084 -1.516 0.658 2023-12-04T05:52:21.002 INFO:teuthology.orchestra.run.smithi195.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 +0.000 0.000 2023-12-04T05:52:21.003 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-04T05:52:21.013 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-04T05:52:21.014 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-04T05:52:21.028 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-04T05:52:21.041 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-04T05:52:21.051 INFO:teuthology.task.internal:Duration was 1883.830750 seconds 2023-12-04T05:52:21.052 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-04T05:52:21.061 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-04T05:52:21.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T05:52:21.064 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T05:52:21.147 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-04T05:52:21.147 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2023-12-04T05:52:21.147 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T05:52:21.157 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi195.front.sepia.ceph.com 2023-12-04T05:52:21.157 DEBUG:teuthology.orchestra.run.smithi195:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T05:52:21.168 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-04T05:52:21.168 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T05:52:21.203 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T05:52:21.233 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-04T05:52:21.233 DEBUG:teuthology.orchestra.run.smithi107:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T05:52:21.266 DEBUG:teuthology.orchestra.run.smithi195:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T05:52:21.328 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-04T05:52:21.338 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-04T05:52:21.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T05:52:21.351 DEBUG:teuthology.orchestra.run.smithi195:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T05:52:21.364 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-04T05:52:21.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T05:52:21.395 DEBUG:teuthology.orchestra.run.smithi195:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T05:52:21.405 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2023-12-04T05:52:21.418 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = core 2023-12-04T05:52:21.442 DEBUG:teuthology.orchestra.run.smithi107:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T05:52:21.465 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi107.front.sepia.ceph.com, flagging run as failed 2023-12-04T05:52:21.465 DEBUG:teuthology.orchestra.run.smithi195:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T05:52:21.488 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi195.front.sepia.ceph.com, flagging run as failed 2023-12-04T05:52:21.488 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-04T05:52:21.500 INFO:teuthology.task.internal:Transferring archived files... 2023-12-04T05:52:21.501 DEBUG:teuthology.misc:Transferring archived files from smithi107:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2023-12-04_05:06:51-fs-wip-vshankar-testing-20231127.102654-distro-default-smithi/7476686/remote/smithi107 2023-12-04T05:52:21.501 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-04T05:52:29.046 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2023-12-04T05:52:29.057 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/vshankar-2023-12-04_05:06:51-fs-wip-vshankar-testing-20231127.102654-distro-default-smithi/7476686/remote/smithi107/coredump/1701668737.44022.core.gz: gzip compressed data, was "1701668737.44022.core", last modified: Mon Dec 4 05:52:21 2023, max compression, original size modulo 2^32 257585152 2023-12-04T05:52:29.057 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2023-12-04T05:52:30.058 INFO:teuthology.task.internal: core looks like: /tmp/tmpconqumxs: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-mds -f --cluster ceph -i a', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/proc/self/exe', platform: 'x86_64' 2023-12-04T05:52:30.086 INFO:teuthology.task.internal: dump_program: ceph-mds -f --cluster ceph -i a 2023-12-04T05:52:30.087 DEBUG:teuthology.orchestra.run.smithi107:> which 'ceph-mds -f --cluster ceph -i a' 2023-12-04T05:52:30.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T05:52:30.094 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 1: "which 'ceph-mds -f --cluster ceph -i a'" 2023-12-04T05:52:30.114 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-04T05:52:30.171 INFO:teuthology.task.internal:Not uploading archives. 2023-12-04T05:52:30.172 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-04T05:52:30.182 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-04T05:52:30.182 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T05:52:30.189 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T05:52:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: 658629 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Dec 4 05:52 /home/ubuntu/cephtest 2023-12-04T05:52:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: 658630 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Dec 4 05:20 /home/ubuntu/cephtest/archive 2023-12-04T05:52:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: 658634 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 05:52 /home/ubuntu/cephtest/archive/syslog 2023-12-04T05:52:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: 658637 4 -rw-rw-rw- 1 ubuntu ubuntu 1058 Dec 4 05:52 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-12-04T05:52:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: 658636 108 -rw-rw-r-- 1 ubuntu ubuntu 106997 Dec 4 05:52 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T05:52:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: 663673 92 -rw-rw-rw- 1 ubuntu ubuntu 93462 Dec 4 05:52 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-12-04T05:52:30.195 INFO:teuthology.orchestra.run.smithi107.stdout: 658631 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 05:45 /home/ubuntu/cephtest/archive/coredump 2023-12-04T05:52:30.195 INFO:teuthology.orchestra.run.smithi107.stdout: 679284 31912 -rw------- 1 root root 257585152 Dec 4 05:45 /home/ubuntu/cephtest/archive/coredump/1701668737.44022.core 2023-12-04T05:52:30.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T05:52:30.196 INFO:teuthology.orchestra.run.smithi107.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-12-04T05:52:30.197 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-12-04T05:52:30.211 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-04T05:52:30.222 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-04T05:52:30.234 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-12-04T05:52:30.234 INFO:teuthology.nuke:Checking targets against current locks 2023-12-04T05:52:30.241 INFO:teuthology.orchestra.run.smithi195.stdout: 655472 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Dec 4 05:52 /home/ubuntu/cephtest 2023-12-04T05:52:30.242 INFO:teuthology.orchestra.run.smithi195.stdout: 658633 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Dec 4 05:20 /home/ubuntu/cephtest/archive 2023-12-04T05:52:30.242 INFO:teuthology.orchestra.run.smithi195.stdout: 658635 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 05:52 /home/ubuntu/cephtest/archive/syslog 2023-12-04T05:52:30.242 INFO:teuthology.orchestra.run.smithi195.stdout: 658639 4 -rw-rw-rw- 1 ubuntu ubuntu 794 Dec 4 05:52 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-12-04T05:52:30.242 INFO:teuthology.orchestra.run.smithi195.stdout: 658636 100 -rw-rw-r-- 1 ubuntu ubuntu 101156 Dec 4 05:52 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T05:52:30.242 INFO:teuthology.orchestra.run.smithi195.stdout: 663680 84 -rw-rw-rw- 1 ubuntu ubuntu 85556 Dec 4 05:52 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-12-04T05:52:30.242 INFO:teuthology.orchestra.run.smithi195.stdout: 658634 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 05:43 /home/ubuntu/cephtest/archive/coredump 2023-12-04T05:52:30.242 INFO:teuthology.orchestra.run.smithi195.stdout: 679081 34500 -rw------- 1 root root 260534272 Dec 4 05:43 /home/ubuntu/cephtest/archive/coredump/1701668621.43769.core 2023-12-04T05:52:30.242 INFO:teuthology.orchestra.run.smithi195.stdout: 679080 36420 -rw------- 1 root root 246202368 Dec 4 05:43 /home/ubuntu/cephtest/archive/coredump/1701668582.43663.core 2023-12-04T05:52:30.243 INFO:teuthology.orchestra.run.smithi195.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-12-04T05:52:30.271 DEBUG:teuthology.nuke:shortname: smithi107 2023-12-04T05:52:30.271 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T05:52:30.295 DEBUG:teuthology.nuke:shortname: smithi195 2023-12-04T05:52:30.296 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T05:52:30.300 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-12-04_05:06:51-fs-wip-vshankar-testing-20231127.102654-distro-default-smithi/7476686', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 05:09:04.580894', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ+YUJ6V8Zs/mK0H0UauHpLsCO+FgY8+C6vlLV1lZ8ShROidb5PMfa1jTYT/99lunhuwhLIQEe6ftNQDd9GOH0I='} 2023-12-04T05:52:30.325 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-12-04_05:06:51-fs-wip-vshankar-testing-20231127.102654-distro-default-smithi/7476686', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 05:09:04.582334', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFxdzf5XteYu1uo6SobO6K4ofYoKZF5RPBVzpjxQRrDSb/joT32TSkCfOjhGbd6T6OSXYnSWiFlLbP63Ry7LmfM='} 2023-12-04T05:52:30.358 INFO:teuthology.orchestra.console.smithi107:Power off 2023-12-04T05:52:30.358 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-04T05:52:30.381 DEBUG:teuthology.orchestra.console.smithi107:power off output: Chassis Power Control: Down/Off 2023-12-04T05:52:30.382 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T05:52:30.399 INFO:teuthology.orchestra.console.smithi195:Power off 2023-12-04T05:52:30.399 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-04T05:52:30.410 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2023-12-04T05:52:30.423 DEBUG:teuthology.orchestra.console.smithi195:power off output: Chassis Power Control: Down/Off 2023-12-04T05:52:30.423 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T05:52:30.450 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2023-12-04T05:52:34.410 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T05:52:34.451 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T05:52:34.522 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2023-12-04T05:52:34.562 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2023-12-04T05:52:38.526 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T05:52:38.563 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T05:52:38.638 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2023-12-04T05:52:38.674 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2023-12-04T05:52:38.740 INFO:teuthology.orchestra.console.smithi107:Power off completed 2023-12-04T05:52:42.678 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T05:52:42.788 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is off 2023-12-04T05:52:42.888 INFO:teuthology.orchestra.console.smithi195:Power off completed 2023-12-04T05:52:42.995 INFO:teuthology.run:Summary data: description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} duration: 1883.8307495117188 failure_reason: 'Command failed (workunit test suites/ffsb.sh) on smithi107 with status 2: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47c1c547372ed5bff7984647dc200bac0e019caf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh''' flavor: default owner: scheduled_vshankar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=00ae224516a3476ea76a04ad4ac52aeb status: fail success: false 2023-12-04T05:52:42.995 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T05:52:43.079 INFO:teuthology.run:FAIL